[PATCH RFC 0/3] Introduce orphan tee_shm and default tee_context

[PATCH RFC 0/3] Introduce orphan tee_shm and default tee_context

[Amirreza Zarrabi]

The TEE subsystem manages three main structures: tee_device, the device
that represents the TEE; tee_context, the context that represents the
TEE client; and tee_shm, which represents the shared memory with the
TEE. When a tee_device is opened, it creates a tee_context instance. The
tee_shm is created for the tee_device when allocating shared memory with
the TEE but is linked to a context. The lifespan of the device is
determined by the presence of context and shared memory, while the
lifespan of a context depends on the client closing the device.

This behavior has been modified, making the lifespan of context
dependent on shared memory. If a client closes the device but doesn’t
release the shared memory, the linked context will remain active,
preventing the release callback from freeing resources in the TEE. This
could lead to a deadlock if the TEE holds a reference to the shared
memory and relies on the release callback to remove the reference.

In this pachset we introduce orphan tee_shm and default tee_context.

When a shared memory becomes orphan because its associated context is
released, it no longer has a tee_context. One method to differentiate
between orphaned and regular shared memory is to use NULL as the linked
context. However, this can cause issues if releasing the shared memory
triggers additional calls, like those to the supplicant, which require a
valid context. Instead of using NULL, an internal tee_context for the
driver can be used.

The driver relies on tee_device_unregister which is a blocking calls
waiting for all context to be released and all shared memory to be freed
before unloading the driver. This means that all contexts, including
internal context, should be closed before tee_device_unregister can
proceed. This can introduce a short window where there is no valid
context to use when releasing the shared memory. The default tee_context
has lifespan similar to the device.

For an orphan tee_shm, default context is used.

This has not been tested. Looking for feedback if this is a reasonable
change.

Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
---
Amirreza Zarrabi (3):
      tee: revert removal of redundant teedev in struct tee_shm
      tee: revert removal of linked list of struct tee_shm
      tee: introduce orphan tee_shm and default context

 drivers/tee/optee/core.c    |  2 +-
 drivers/tee/optee/ffa_abi.c |  2 +-
 drivers/tee/optee/smc_abi.c |  2 +-
 drivers/tee/tee_core.c      | 84 +++++++++++++++++++++++++++++----------------
 drivers/tee/tee_private.h   |  3 --
 drivers/tee/tee_shm.c       | 41 ++++++++++++----------
 include/linux/tee_core.h    | 15 ++++++++
 include/linux/tee_drv.h     | 13 ++++---
 8 files changed, 100 insertions(+), 62 deletions(-)
---
base-commit: ae58226b89ac0cffa05ba7357733776542e40216
change-id: 20241120-fix-tee_shm-refcount-upstream-c671b89fbe67

Best regards,
-- 
Amirreza Zarrabi <quic_azarrabi@quicinc.com>

[PATCH RFC 1/3] tee: revert removal of redundant teedev in struct tee_shm

[Amirreza Zarrabi]

In commit 5271b2011e44 ("tee: remove redundant teedev in struct tee_shm"),
the reference to teedev was removed following the change in
commit 217e0250cccb ("tee: use reference counting for tee_context").
This change ensured that the ctx in tee_shm remains valid as long as the
shared buffer is valid, and teedev is accessible from ctx. It made teedev
in tee_shm redundant.

Reintroduce teedev to tee_shm to facilitate the introduction of orphan
shared memory, which may not be linked to the context it was originally
associated with.

Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
---
 drivers/tee/tee_shm.c   | 10 +++++++---
 include/linux/tee_drv.h |  2 ++
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/drivers/tee/tee_shm.c b/drivers/tee/tee_shm.c
index daf6e5cfd59a..31e032446cf5 100644
--- a/drivers/tee/tee_shm.c
+++ b/drivers/tee/tee_shm.c
@@ -43,8 +43,10 @@ static void release_registered_pages(struct tee_shm *shm)
 	}
 }
 
-static void tee_shm_release(struct tee_device *teedev, struct tee_shm *shm)
+static void tee_shm_release(struct tee_shm *shm)
 {
+	struct tee_device *teedev = shm->teedev;
+
 	if (shm->flags & TEE_SHM_POOL) {
 		teedev->pool->ops->free(teedev->pool, shm);
 	} else if (shm->flags & TEE_SHM_DYNAMIC) {
@@ -89,6 +91,7 @@ static struct tee_shm *shm_alloc_helper(struct tee_context *ctx, size_t size,
 
 	refcount_set(&shm->refcount, 1);
 	shm->flags = flags;
+	shm->teedev = teedev;
 	shm->id = id;
 
 	/*
@@ -298,6 +301,7 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
 
 	refcount_set(&shm->refcount, 1);
 	shm->flags = flags;
+	shm->teedev = teedev;
 	shm->ctx = ctx;
 	shm->id = id;
 	addr = untagged_addr((unsigned long)iter_iov_addr(iter));
@@ -560,7 +564,7 @@ EXPORT_SYMBOL_GPL(tee_shm_get_from_id);
  */
 void tee_shm_put(struct tee_shm *shm)
 {
-	struct tee_device *teedev = shm->ctx->teedev;
+	struct tee_device *teedev = shm->teedev;
 	bool do_release = false;
 
 	mutex_lock(&teedev->mutex);
@@ -578,6 +582,6 @@ void tee_shm_put(struct tee_shm *shm)
 	mutex_unlock(&teedev->mutex);
 
 	if (do_release)
-		tee_shm_release(teedev, shm);
+		tee_shm_release(shm);
 }
 EXPORT_SYMBOL_GPL(tee_shm_put);
diff --git a/include/linux/tee_drv.h b/include/linux/tee_drv.h
index a54c203000ed..56560441b2cb 100644
--- a/include/linux/tee_drv.h
+++ b/include/linux/tee_drv.h
@@ -47,6 +47,7 @@ struct tee_context {
 
 /**
  * struct tee_shm - shared memory object
+ * @teedev:	device used to allocate the object
  * @ctx:	context using the object
  * @paddr:	physical address of the shared memory
  * @kaddr:	virtual address of the shared memory
@@ -63,6 +64,7 @@ struct tee_context {
  *		used by all drivers
  */
 struct tee_shm {
+	struct tee_device *teedev;
 	struct tee_context *ctx;
 	phys_addr_t paddr;
 	void *kaddr;

-- 
2.34.1

[PATCH RFC 2/3] tee: revert removal of linked list of struct tee_shm

[Amirreza Zarrabi]

Partially revert commit 59a135f6fb66 ("tee: remove linked list of
struct tee_shm"). Reintroduce the linked list to track all tee_shm
instances associated with a context.

Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
---
 drivers/tee/tee_core.c  |  1 +
 drivers/tee/tee_shm.c   | 13 +++++++++++++
 include/linux/tee_drv.h |  4 ++++
 3 files changed, 18 insertions(+)

diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
index d113679b1e2d..93f3b330aec8 100644
--- a/drivers/tee/tee_core.c
+++ b/drivers/tee/tee_core.c
@@ -59,6 +59,7 @@ struct tee_context *teedev_open(struct tee_device *teedev)
 
 	kref_init(&ctx->refcount);
 	ctx->teedev = teedev;
+	INIT_LIST_HEAD(&ctx->list_shm);
 	rc = teedev->desc->ops->open(ctx);
 	if (rc)
 		goto err;
diff --git a/drivers/tee/tee_shm.c b/drivers/tee/tee_shm.c
index 31e032446cf5..c0164c0f4a01 100644
--- a/drivers/tee/tee_shm.c
+++ b/drivers/tee/tee_shm.c
@@ -108,6 +108,10 @@ static struct tee_shm *shm_alloc_helper(struct tee_context *ctx, size_t size,
 		goto err_kfree;
 	}
 
+	mutex_lock(&teedev->mutex);
+	list_add_tail(&shm->link, &ctx->list_shm);
+	mutex_unlock(&teedev->mutex);
+
 	teedev_ctx_get(ctx);
 	return shm;
 err_kfree:
@@ -343,6 +347,10 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
 		goto err_put_shm_pages;
 	}
 
+	mutex_lock(&teedev->mutex);
+	list_add_tail(&shm->link, &ctx->list_shm);
+	mutex_unlock(&teedev->mutex);
+
 	return shm;
 err_put_shm_pages:
 	if (!iov_iter_is_kvec(iter))
@@ -577,6 +585,11 @@ void tee_shm_put(struct tee_shm *shm)
 		 */
 		if (shm->id >= 0)
 			idr_remove(&teedev->idr, shm->id);
+
+		/* The context owns shm may be gone already. */
+		if (shm->ctx)
+			list_del(&shm->link);
+
 		do_release = true;
 	}
 	mutex_unlock(&teedev->mutex);
diff --git a/include/linux/tee_drv.h b/include/linux/tee_drv.h
index 56560441b2cb..1b57cddfecc8 100644
--- a/include/linux/tee_drv.h
+++ b/include/linux/tee_drv.h
@@ -23,6 +23,7 @@ struct tee_device;
 /**
  * struct tee_context - driver specific context on file pointer data
  * @teedev:	pointer to this drivers struct tee_device
+ * @list_shm:	List of shared memory object owned by this context
  * @data:	driver specific context data, managed by the driver
  * @refcount:	reference counter for this structure
  * @releasing:  flag that indicates if context is being released right now.
@@ -38,6 +39,7 @@ struct tee_device;
  */
 struct tee_context {
 	struct tee_device *teedev;
+	struct list_head list_shm;
 	void *data;
 	struct kref refcount;
 	bool releasing;
@@ -49,6 +51,7 @@ struct tee_context {
  * struct tee_shm - shared memory object
  * @teedev:	device used to allocate the object
  * @ctx:	context using the object
+ * @link	link element
  * @paddr:	physical address of the shared memory
  * @kaddr:	virtual address of the shared memory
  * @size:	size of shared memory
@@ -66,6 +69,7 @@ struct tee_context {
 struct tee_shm {
 	struct tee_device *teedev;
 	struct tee_context *ctx;
+	struct list_head link;
 	phys_addr_t paddr;
 	void *kaddr;
 	size_t size;

-- 
2.34.1

[PATCH RFC 3/3] tee: introduce orphan tee_shm and default context

[Amirreza Zarrabi]

The default context has a lifespan similar to the tee_device.
It is used as a context for shared memory if the context to which the
shared memory belongs is released, making the tee_shm an orphan.
This allows the driver implementing shm_unregister to safely make
subsequent calls, such as to a supplicant if needed.

It also enables users to free the shared memory while the driver is
blocked on unregister_tee_device safely.

Preferably, this should be used for all driver internal uses, using
teedev_get_def_context rather than calling teedev_open.

Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
---
 drivers/tee/optee/core.c    |  2 +-
 drivers/tee/optee/ffa_abi.c |  2 +-
 drivers/tee/optee/smc_abi.c |  2 +-
 drivers/tee/tee_core.c      | 83 +++++++++++++++++++++++++++++----------------
 drivers/tee/tee_private.h   |  3 --
 drivers/tee/tee_shm.c       | 18 ++--------
 include/linux/tee_core.h    | 15 ++++++++
 include/linux/tee_drv.h     |  7 ----
 8 files changed, 73 insertions(+), 59 deletions(-)

diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
index c75fddc83576..78d43d0c8014 100644
--- a/drivers/tee/optee/core.c
+++ b/drivers/tee/optee/core.c
@@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
 
 	optee_notif_uninit(optee);
 	optee_shm_arg_cache_uninit(optee);
-	teedev_close_context(optee->ctx);
+
 	/*
 	 * The two devices have to be unregistered before we can free the
 	 * other resources.
diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
index f3af5666bb11..6ad94f0788ad 100644
--- a/drivers/tee/optee/ffa_abi.c
+++ b/drivers/tee/optee/ffa_abi.c
@@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
 	optee_shm_arg_cache_init(optee, arg_cache_flags);
 	mutex_init(&optee->rpmb_dev_mutex);
 	ffa_dev_set_drvdata(ffa_dev, optee);
-	ctx = teedev_open(optee->teedev);
+	ctx = teedev_get_def_context(optee->teedev);
 	if (IS_ERR(ctx)) {
 		rc = PTR_ERR(ctx);
 		goto err_rhashtable_free;
diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
index e9456e3e74cc..c77a3e631d04 100644
--- a/drivers/tee/optee/smc_abi.c
+++ b/drivers/tee/optee/smc_abi.c
@@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
 	mutex_init(&optee->rpmb_dev_mutex);
 
 	platform_set_drvdata(pdev, optee);
-	ctx = teedev_open(optee->teedev);
+	ctx = teedev_get_def_context(optee->teedev);
 	if (IS_ERR(ctx)) {
 		rc = PTR_ERR(ctx);
 		goto err_supp_uninit;
diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
index 93f3b330aec8..805e1336089d 100644
--- a/drivers/tee/tee_core.c
+++ b/drivers/tee/tee_core.c
@@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
 		goto err;
 	}
 
-	kref_init(&ctx->refcount);
 	ctx->teedev = teedev;
 	INIT_LIST_HEAD(&ctx->list_shm);
 	rc = teedev->desc->ops->open(ctx);
@@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
 }
 EXPORT_SYMBOL_GPL(teedev_open);
 
-void teedev_ctx_get(struct tee_context *ctx)
+struct tee_context *teedev_get_def_context(struct tee_device *teedev)
 {
-	if (ctx->releasing)
-		return;
+	int rc;
+	struct tee_context *ctx = &teedev->def_ctx;
 
-	kref_get(&ctx->refcount);
-}
+	ctx->teedev = teedev;
+	INIT_LIST_HEAD(&ctx->list_shm);
+	rc = teedev->desc->ops->open(ctx);
+	if (rc)
+		return ERR_PTR(rc);
 
-static void teedev_ctx_release(struct kref *ref)
-{
-	struct tee_context *ctx = container_of(ref, struct tee_context,
-					       refcount);
-	ctx->releasing = true;
-	ctx->teedev->desc->ops->release(ctx);
-	kfree(ctx);
+	return ctx;
 }
+EXPORT_SYMBOL_GPL(teedev_get_def_context);
 
-void teedev_ctx_put(struct tee_context *ctx)
+void teedev_close_context(struct tee_context *ctx)
 {
-	if (ctx->releasing)
+	struct tee_device *teedev = ctx->teedev;
+	struct tee_shm *shm;
+
+	if (ctx == &teedev->def_ctx)
 		return;
 
-	kref_put(&ctx->refcount, teedev_ctx_release);
-}
+	teedev->desc->ops->release(ctx);
 
-void teedev_close_context(struct tee_context *ctx)
-{
-	struct tee_device *teedev = ctx->teedev;
+	mutex_lock(&teedev->mutex);
+	list_for_each_entry(shm, &ctx->list_shm, link) {
+		/* Context released. However, shm still holding a teedev reference.
+		 * Replace shm->ctx with the default context so that tee_shm_get_from_id()
+		 * fails (i.e. it is not accessible from userspace) but shm still
+		 * holds a valid context for further clean up, e.g. shm_unregister().
+		 */
+		shm->ctx = &teedev->def_ctx;
+	}
+	mutex_unlock(&teedev->mutex);
 
-	teedev_ctx_put(ctx);
+	kfree(ctx);
 	tee_device_put(teedev);
 }
 EXPORT_SYMBOL_GPL(teedev_close_context);
@@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
 
 	teedev->desc = teedesc;
 	teedev->pool = pool;
+	/* Only open default context when teedev_get_def_context() called. */
+	teedev->def_ctx.teedev = NULL;
 
 	return teedev;
 err_devt:
@@ -1027,16 +1035,31 @@ EXPORT_SYMBOL_GPL(tee_device_register);
 
 void tee_device_put(struct tee_device *teedev)
 {
-	mutex_lock(&teedev->mutex);
-	/* Shouldn't put in this state */
-	if (!WARN_ON(!teedev->desc)) {
-		teedev->num_users--;
-		if (!teedev->num_users) {
-			teedev->desc = NULL;
-			complete(&teedev->c_no_users);
-		}
+	const struct tee_desc *desc;
+
+	scoped_guard(mutex, &teedev->mutex) {
+		desc = teedev->desc;
+
+		/* Shouldn't put in this state */
+		if (WARN_ON(!desc))
+			return;
+
+		/* If there is still users for teedev */
+		if (--teedev->num_users)
+			return;
+
+		/* tee_device_unregister() has been called and there is no
+		 * user in userspace or kernel, including orphan shm for teedev.
+		 * Set teedev->desc to NULL, so that teedev can not be reused.
+		 */
+		teedev->desc = NULL;
 	}
-	mutex_unlock(&teedev->mutex);
+
+	/* Release the default context */
+	desc->ops->release(&teedev->def_ctx);
+	teedev->def_ctx.teedev = NULL;
+
+	complete(&teedev->c_no_users);
 }
 
 bool tee_device_get(struct tee_device *teedev)
diff --git a/drivers/tee/tee_private.h b/drivers/tee/tee_private.h
index 9bc50605227c..6c7bcc308958 100644
--- a/drivers/tee/tee_private.h
+++ b/drivers/tee/tee_private.h
@@ -17,9 +17,6 @@ int tee_shm_get_fd(struct tee_shm *shm);
 bool tee_device_get(struct tee_device *teedev);
 void tee_device_put(struct tee_device *teedev);
 
-void teedev_ctx_get(struct tee_context *ctx);
-void teedev_ctx_put(struct tee_context *ctx);
-
 struct tee_shm *tee_shm_alloc_user_buf(struct tee_context *ctx, size_t size);
 struct tee_shm *tee_shm_register_user_buf(struct tee_context *ctx,
 					  unsigned long addr, size_t length);
diff --git a/drivers/tee/tee_shm.c b/drivers/tee/tee_shm.c
index c0164c0f4a01..f07274291edf 100644
--- a/drivers/tee/tee_shm.c
+++ b/drivers/tee/tee_shm.c
@@ -59,8 +59,6 @@ static void tee_shm_release(struct tee_shm *shm)
 		release_registered_pages(shm);
 	}
 
-	teedev_ctx_put(shm->ctx);
-
 	kfree(shm);
 
 	tee_device_put(teedev);
@@ -93,13 +91,6 @@ static struct tee_shm *shm_alloc_helper(struct tee_context *ctx, size_t size,
 	shm->flags = flags;
 	shm->teedev = teedev;
 	shm->id = id;
-
-	/*
-	 * We're assigning this as it is needed if the shm is to be
-	 * registered. If this function returns OK then the caller expected
-	 * to call teedev_ctx_get() or clear shm->ctx in case it's not
-	 * needed any longer.
-	 */
 	shm->ctx = ctx;
 
 	rc = teedev->pool->ops->alloc(teedev->pool, shm, size, align);
@@ -112,7 +103,6 @@ static struct tee_shm *shm_alloc_helper(struct tee_context *ctx, size_t size,
 	list_add_tail(&shm->link, &ctx->list_shm);
 	mutex_unlock(&teedev->mutex);
 
-	teedev_ctx_get(ctx);
 	return shm;
 err_kfree:
 	kfree(shm);
@@ -295,12 +285,10 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
 		goto err_dev_put;
 	}
 
-	teedev_ctx_get(ctx);
-
 	shm = kzalloc(sizeof(*shm), GFP_KERNEL);
 	if (!shm) {
 		ret = ERR_PTR(-ENOMEM);
-		goto err_ctx_put;
+		goto err_dev_put;
 	}
 
 	refcount_set(&shm->refcount, 1);
@@ -313,7 +301,7 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
 	num_pages = iov_iter_npages(iter, INT_MAX);
 	if (!num_pages) {
 		ret = ERR_PTR(-ENOMEM);
-		goto err_ctx_put;
+		goto err_dev_put;
 	}
 
 	shm->pages = kcalloc(num_pages, sizeof(*shm->pages), GFP_KERNEL);
@@ -361,8 +349,6 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
 	kfree(shm->pages);
 err_free_shm:
 	kfree(shm);
-err_ctx_put:
-	teedev_ctx_put(ctx);
 err_dev_put:
 	tee_device_put(teedev);
 	return ret;
diff --git a/include/linux/tee_core.h b/include/linux/tee_core.h
index a38494d6b5f4..13393ddac530 100644
--- a/include/linux/tee_core.h
+++ b/include/linux/tee_core.h
@@ -44,6 +44,7 @@
  * @idr:	register of user space shared memory objects allocated or
  *		registered on this device
  * @pool:	shared memory pool
+ * @def_ctx:	default context used if there is no context available, e.g. internal driver calls.
  */
 struct tee_device {
 	char name[TEE_MAX_DEV_NAME_LEN];
@@ -60,6 +61,7 @@ struct tee_device {
 
 	struct idr idr;
 	struct tee_shm_pool *pool;
+	struct tee_context def_ctx;
 };
 
 /**
@@ -309,6 +311,19 @@ static inline bool tee_param_is_memref(struct tee_param *param)
  */
 struct tee_context *teedev_open(struct tee_device *teedev);
 
+/**
+ * teedev_get_def_context() - Get default context for a struct tee_device
+ * @teedev:	Device to open
+ *
+ * Unlike a context that returned from teedev_open(), the default context is static
+ * and available as long as @teedev has a user ''other then this context''. This context
+ * can be used for driver internal operation and clean up where a context should be
+ * available, while tee_device_unregister() is waiting for other users to go away.
+ *
+ * @return a pointer to struct tee_context on success or an ERR_PTR on failure.
+ */
+struct tee_context *teedev_get_def_context(struct tee_device *teedev);
+
 /**
  * teedev_close_context() - closes a struct tee_context
  * @ctx:	The struct tee_context to close
diff --git a/include/linux/tee_drv.h b/include/linux/tee_drv.h
index 1b57cddfecc8..9633e14ba484 100644
--- a/include/linux/tee_drv.h
+++ b/include/linux/tee_drv.h
@@ -7,7 +7,6 @@
 #define __TEE_DRV_H
 
 #include <linux/device.h>
-#include <linux/kref.h>
 #include <linux/list.h>
 #include <linux/mod_devicetable.h>
 #include <linux/tee.h>
@@ -25,10 +24,6 @@ struct tee_device;
  * @teedev:	pointer to this drivers struct tee_device
  * @list_shm:	List of shared memory object owned by this context
  * @data:	driver specific context data, managed by the driver
- * @refcount:	reference counter for this structure
- * @releasing:  flag that indicates if context is being released right now.
- *		It is needed to break circular dependency on context during
- *              shared memory release.
  * @supp_nowait: flag that indicates that requests in this context should not
  *              wait for tee-supplicant daemon to be started if not present
  *              and just return with an error code. It is needed for requests
@@ -41,8 +36,6 @@ struct tee_context {
 	struct tee_device *teedev;
 	struct list_head list_shm;
 	void *data;
-	struct kref refcount;
-	bool releasing;
 	bool supp_nowait;
 	bool cap_memref_null;
 };

-- 
2.34.1

Re: [PATCH RFC 3/3] tee: introduce orphan tee_shm and default context

[Jens Wiklander]

Hi Amirreza,

On Thu, Nov 21, 2024 at 2:37 AM Amirreza Zarrabi
<quic_azarrabi@quicinc.com> wrote:
>
> The default context has a lifespan similar to the tee_device.
> It is used as a context for shared memory if the context to which the
> shared memory belongs is released, making the tee_shm an orphan.
> This allows the driver implementing shm_unregister to safely make
> subsequent calls, such as to a supplicant if needed.
>
> It also enables users to free the shared memory while the driver is
> blocked on unregister_tee_device safely.
>
> Preferably, this should be used for all driver internal uses, using
> teedev_get_def_context rather than calling teedev_open.
>
> Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
> ---
>  drivers/tee/optee/core.c    |  2 +-
>  drivers/tee/optee/ffa_abi.c |  2 +-
>  drivers/tee/optee/smc_abi.c |  2 +-
>  drivers/tee/tee_core.c      | 83 +++++++++++++++++++++++++++++----------------
>  drivers/tee/tee_private.h   |  3 --
>  drivers/tee/tee_shm.c       | 18 ++--------
>  include/linux/tee_core.h    | 15 ++++++++
>  include/linux/tee_drv.h     |  7 ----
>  8 files changed, 73 insertions(+), 59 deletions(-)
>
> diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
> index c75fddc83576..78d43d0c8014 100644
> --- a/drivers/tee/optee/core.c
> +++ b/drivers/tee/optee/core.c
> @@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
>
>         optee_notif_uninit(optee);
>         optee_shm_arg_cache_uninit(optee);
> -       teedev_close_context(optee->ctx);
> +
>         /*
>          * The two devices have to be unregistered before we can free the
>          * other resources.
> diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
> index f3af5666bb11..6ad94f0788ad 100644
> --- a/drivers/tee/optee/ffa_abi.c
> +++ b/drivers/tee/optee/ffa_abi.c
> @@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
>         optee_shm_arg_cache_init(optee, arg_cache_flags);
>         mutex_init(&optee->rpmb_dev_mutex);
>         ffa_dev_set_drvdata(ffa_dev, optee);
> -       ctx = teedev_open(optee->teedev);
> +       ctx = teedev_get_def_context(optee->teedev);
>         if (IS_ERR(ctx)) {
>                 rc = PTR_ERR(ctx);
>                 goto err_rhashtable_free;
> diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
> index e9456e3e74cc..c77a3e631d04 100644
> --- a/drivers/tee/optee/smc_abi.c
> +++ b/drivers/tee/optee/smc_abi.c
> @@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
>         mutex_init(&optee->rpmb_dev_mutex);
>
>         platform_set_drvdata(pdev, optee);
> -       ctx = teedev_open(optee->teedev);
> +       ctx = teedev_get_def_context(optee->teedev);
>         if (IS_ERR(ctx)) {
>                 rc = PTR_ERR(ctx);
>                 goto err_supp_uninit;
> diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
> index 93f3b330aec8..805e1336089d 100644
> --- a/drivers/tee/tee_core.c
> +++ b/drivers/tee/tee_core.c
> @@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
>                 goto err;
>         }
>
> -       kref_init(&ctx->refcount);
>         ctx->teedev = teedev;
>         INIT_LIST_HEAD(&ctx->list_shm);
>         rc = teedev->desc->ops->open(ctx);
> @@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
>  }
>  EXPORT_SYMBOL_GPL(teedev_open);
>
> -void teedev_ctx_get(struct tee_context *ctx)
> +struct tee_context *teedev_get_def_context(struct tee_device *teedev)
>  {
> -       if (ctx->releasing)
> -               return;
> +       int rc;
> +       struct tee_context *ctx = &teedev->def_ctx;
>
> -       kref_get(&ctx->refcount);
> -}
> +       ctx->teedev = teedev;
> +       INIT_LIST_HEAD(&ctx->list_shm);
> +       rc = teedev->desc->ops->open(ctx);
> +       if (rc)
> +               return ERR_PTR(rc);

I think ctx->teedev and ctx->list_shm must always be initialized or
&teedev->def_ctx can't be used in teedev_close_context().
We could initialize teedev->def_ctx on the first call to teedev_open()
on that tee_device. We need a way to tell the
teedev->desc->ops->open() to the backed driver that it's initializing
the default context though, or optee_open() can't handle the
tee-supplicant case properly.

Should we allow this function to be called more than once for each teedev?
Do we need serialization in this function if it's called after the
driver is probed?

>
> -static void teedev_ctx_release(struct kref *ref)
> -{
> -       struct tee_context *ctx = container_of(ref, struct tee_context,
> -                                              refcount);
> -       ctx->releasing = true;
> -       ctx->teedev->desc->ops->release(ctx);
> -       kfree(ctx);
> +       return ctx;
>  }
> +EXPORT_SYMBOL_GPL(teedev_get_def_context);
>
> -void teedev_ctx_put(struct tee_context *ctx)
> +void teedev_close_context(struct tee_context *ctx)
>  {
> -       if (ctx->releasing)
> +       struct tee_device *teedev = ctx->teedev;
> +       struct tee_shm *shm;
> +
> +       if (ctx == &teedev->def_ctx)
>                 return;
>
> -       kref_put(&ctx->refcount, teedev_ctx_release);
> -}
> +       teedev->desc->ops->release(ctx);
>
> -void teedev_close_context(struct tee_context *ctx)
> -{
> -       struct tee_device *teedev = ctx->teedev;
> +       mutex_lock(&teedev->mutex);
> +       list_for_each_entry(shm, &ctx->list_shm, link) {
> +               /* Context released. However, shm still holding a teedev reference.
> +                * Replace shm->ctx with the default context so that tee_shm_get_from_id()
> +                * fails (i.e. it is not accessible from userspace) but shm still
> +                * holds a valid context for further clean up, e.g. shm_unregister().
> +                */

/*
 * Please format
 * multiline comments
 * like this. Please
 * keep the lines at
 * max 80 columns
 * here and at other
 * places in the patch-
 * set.
 */

> +               shm->ctx = &teedev->def_ctx;

shm->ctx will always point to a valid context, even if it is the
default context. It seems that we can always get hold of the correct
teedev via shm->ctx->teedev. Do we need "tee: revert removal of
redundant teedev in struct tee_shm"?

Shouldn't the shm be removed from the ctx->list_shm and be moved to
teedev->def_ctx.list_shm?

> +       }
> +       mutex_unlock(&teedev->mutex);
>
> -       teedev_ctx_put(ctx);
> +       kfree(ctx);
>         tee_device_put(teedev);
>  }
>  EXPORT_SYMBOL_GPL(teedev_close_context);
> @@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
>
>         teedev->desc = teedesc;
>         teedev->pool = pool;
> +       /* Only open default context when teedev_get_def_context() called. */
> +       teedev->def_ctx.teedev = NULL;
>
>         return teedev;
>  err_devt:
> @@ -1027,16 +1035,31 @@ EXPORT_SYMBOL_GPL(tee_device_register);
>
>  void tee_device_put(struct tee_device *teedev)
>  {
> -       mutex_lock(&teedev->mutex);
> -       /* Shouldn't put in this state */
> -       if (!WARN_ON(!teedev->desc)) {
> -               teedev->num_users--;
> -               if (!teedev->num_users) {
> -                       teedev->desc = NULL;
> -                       complete(&teedev->c_no_users);
> -               }
> +       const struct tee_desc *desc;
> +
> +       scoped_guard(mutex, &teedev->mutex) {
> +               desc = teedev->desc;
> +
> +               /* Shouldn't put in this state */
> +               if (WARN_ON(!desc))
> +                       return;
> +
> +               /* If there is still users for teedev */
> +               if (--teedev->num_users)

Please do teedev->num_users-- first and then check. It makes the code
easier to read.

> +                       return;
> +
> +               /* tee_device_unregister() has been called and there is no
> +                * user in userspace or kernel, including orphan shm for teedev.
> +                * Set teedev->desc to NULL, so that teedev can not be reused.
> +                */
> +               teedev->desc = NULL;
>         }
> -       mutex_unlock(&teedev->mutex);
> +
> +       /* Release the default context */
> +       desc->ops->release(&teedev->def_ctx);

This should only be done if teedev->def_ctx has been initialized.

Cheers,
Jens

> +       teedev->def_ctx.teedev = NULL;
> +
> +       complete(&teedev->c_no_users);
>  }
>
>  bool tee_device_get(struct tee_device *teedev)
> diff --git a/drivers/tee/tee_private.h b/drivers/tee/tee_private.h
> index 9bc50605227c..6c7bcc308958 100644
> --- a/drivers/tee/tee_private.h
> +++ b/drivers/tee/tee_private.h
> @@ -17,9 +17,6 @@ int tee_shm_get_fd(struct tee_shm *shm);
>  bool tee_device_get(struct tee_device *teedev);
>  void tee_device_put(struct tee_device *teedev);
>
> -void teedev_ctx_get(struct tee_context *ctx);
> -void teedev_ctx_put(struct tee_context *ctx);
> -
>  struct tee_shm *tee_shm_alloc_user_buf(struct tee_context *ctx, size_t size);
>  struct tee_shm *tee_shm_register_user_buf(struct tee_context *ctx,
>                                           unsigned long addr, size_t length);
> diff --git a/drivers/tee/tee_shm.c b/drivers/tee/tee_shm.c
> index c0164c0f4a01..f07274291edf 100644
> --- a/drivers/tee/tee_shm.c
> +++ b/drivers/tee/tee_shm.c
> @@ -59,8 +59,6 @@ static void tee_shm_release(struct tee_shm *shm)
>                 release_registered_pages(shm);
>         }
>
> -       teedev_ctx_put(shm->ctx);
> -
>         kfree(shm);
>
>         tee_device_put(teedev);
> @@ -93,13 +91,6 @@ static struct tee_shm *shm_alloc_helper(struct tee_context *ctx, size_t size,
>         shm->flags = flags;
>         shm->teedev = teedev;
>         shm->id = id;
> -
> -       /*
> -        * We're assigning this as it is needed if the shm is to be
> -        * registered. If this function returns OK then the caller expected
> -        * to call teedev_ctx_get() or clear shm->ctx in case it's not
> -        * needed any longer.
> -        */
>         shm->ctx = ctx;
>
>         rc = teedev->pool->ops->alloc(teedev->pool, shm, size, align);
> @@ -112,7 +103,6 @@ static struct tee_shm *shm_alloc_helper(struct tee_context *ctx, size_t size,
>         list_add_tail(&shm->link, &ctx->list_shm);
>         mutex_unlock(&teedev->mutex);
>
> -       teedev_ctx_get(ctx);
>         return shm;
>  err_kfree:
>         kfree(shm);
> @@ -295,12 +285,10 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
>                 goto err_dev_put;
>         }
>
> -       teedev_ctx_get(ctx);
> -
>         shm = kzalloc(sizeof(*shm), GFP_KERNEL);
>         if (!shm) {
>                 ret = ERR_PTR(-ENOMEM);
> -               goto err_ctx_put;
> +               goto err_dev_put;
>         }
>
>         refcount_set(&shm->refcount, 1);
> @@ -313,7 +301,7 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
>         num_pages = iov_iter_npages(iter, INT_MAX);
>         if (!num_pages) {
>                 ret = ERR_PTR(-ENOMEM);
> -               goto err_ctx_put;
> +               goto err_dev_put;
>         }
>
>         shm->pages = kcalloc(num_pages, sizeof(*shm->pages), GFP_KERNEL);
> @@ -361,8 +349,6 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
>         kfree(shm->pages);
>  err_free_shm:
>         kfree(shm);
> -err_ctx_put:
> -       teedev_ctx_put(ctx);
>  err_dev_put:
>         tee_device_put(teedev);
>         return ret;
> diff --git a/include/linux/tee_core.h b/include/linux/tee_core.h
> index a38494d6b5f4..13393ddac530 100644
> --- a/include/linux/tee_core.h
> +++ b/include/linux/tee_core.h
> @@ -44,6 +44,7 @@
>   * @idr:       register of user space shared memory objects allocated or
>   *             registered on this device
>   * @pool:      shared memory pool
> + * @def_ctx:   default context used if there is no context available, e.g. internal driver calls.
>   */
>  struct tee_device {
>         char name[TEE_MAX_DEV_NAME_LEN];
> @@ -60,6 +61,7 @@ struct tee_device {
>
>         struct idr idr;
>         struct tee_shm_pool *pool;
> +       struct tee_context def_ctx;
>  };
>
>  /**
> @@ -309,6 +311,19 @@ static inline bool tee_param_is_memref(struct tee_param *param)
>   */
>  struct tee_context *teedev_open(struct tee_device *teedev);
>
> +/**
> + * teedev_get_def_context() - Get default context for a struct tee_device
> + * @teedev:    Device to open
> + *
> + * Unlike a context that returned from teedev_open(), the default context is static
> + * and available as long as @teedev has a user ''other then this context''. This context
> + * can be used for driver internal operation and clean up where a context should be
> + * available, while tee_device_unregister() is waiting for other users to go away.
> + *
> + * @return a pointer to struct tee_context on success or an ERR_PTR on failure.
> + */
> +struct tee_context *teedev_get_def_context(struct tee_device *teedev);
> +
>  /**
>   * teedev_close_context() - closes a struct tee_context
>   * @ctx:       The struct tee_context to close
> diff --git a/include/linux/tee_drv.h b/include/linux/tee_drv.h
> index 1b57cddfecc8..9633e14ba484 100644
> --- a/include/linux/tee_drv.h
> +++ b/include/linux/tee_drv.h
> @@ -7,7 +7,6 @@
>  #define __TEE_DRV_H
>
>  #include <linux/device.h>
> -#include <linux/kref.h>
>  #include <linux/list.h>
>  #include <linux/mod_devicetable.h>
>  #include <linux/tee.h>
> @@ -25,10 +24,6 @@ struct tee_device;
>   * @teedev:    pointer to this drivers struct tee_device
>   * @list_shm:  List of shared memory object owned by this context
>   * @data:      driver specific context data, managed by the driver
> - * @refcount:  reference counter for this structure
> - * @releasing:  flag that indicates if context is being released right now.
> - *             It is needed to break circular dependency on context during
> - *              shared memory release.
>   * @supp_nowait: flag that indicates that requests in this context should not
>   *              wait for tee-supplicant daemon to be started if not present
>   *              and just return with an error code. It is needed for requests
> @@ -41,8 +36,6 @@ struct tee_context {
>         struct tee_device *teedev;
>         struct list_head list_shm;
>         void *data;
> -       struct kref refcount;
> -       bool releasing;
>         bool supp_nowait;
>         bool cap_memref_null;
>  };
>
> --
> 2.34.1
>

Re: [PATCH RFC 3/3] tee: introduce orphan tee_shm and default context

[Amirreza Zarrabi]

On 11/21/2024 11:08 PM, Jens Wiklander wrote:

Hi Jens,

> Hi Amirreza,
> 
> On Thu, Nov 21, 2024 at 2:37 AM Amirreza Zarrabi
> <quic_azarrabi@quicinc.com> wrote:
>>
>> The default context has a lifespan similar to the tee_device.
>> It is used as a context for shared memory if the context to which the
>> shared memory belongs is released, making the tee_shm an orphan.
>> This allows the driver implementing shm_unregister to safely make
>> subsequent calls, such as to a supplicant if needed.
>>
>> It also enables users to free the shared memory while the driver is
>> blocked on unregister_tee_device safely.
>>
>> Preferably, this should be used for all driver internal uses, using
>> teedev_get_def_context rather than calling teedev_open.
>>
>> Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
>> ---
>>  drivers/tee/optee/core.c    |  2 +-
>>  drivers/tee/optee/ffa_abi.c |  2 +-
>>  drivers/tee/optee/smc_abi.c |  2 +-
>>  drivers/tee/tee_core.c      | 83 +++++++++++++++++++++++++++++----------------
>>  drivers/tee/tee_private.h   |  3 --
>>  drivers/tee/tee_shm.c       | 18 ++--------
>>  include/linux/tee_core.h    | 15 ++++++++
>>  include/linux/tee_drv.h     |  7 ----
>>  8 files changed, 73 insertions(+), 59 deletions(-)
>>
>> diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
>> index c75fddc83576..78d43d0c8014 100644
>> --- a/drivers/tee/optee/core.c
>> +++ b/drivers/tee/optee/core.c
>> @@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
>>
>>         optee_notif_uninit(optee);
>>         optee_shm_arg_cache_uninit(optee);
>> -       teedev_close_context(optee->ctx);
>> +
>>         /*
>>          * The two devices have to be unregistered before we can free the
>>          * other resources.
>> diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
>> index f3af5666bb11..6ad94f0788ad 100644
>> --- a/drivers/tee/optee/ffa_abi.c
>> +++ b/drivers/tee/optee/ffa_abi.c
>> @@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
>>         optee_shm_arg_cache_init(optee, arg_cache_flags);
>>         mutex_init(&optee->rpmb_dev_mutex);
>>         ffa_dev_set_drvdata(ffa_dev, optee);
>> -       ctx = teedev_open(optee->teedev);
>> +       ctx = teedev_get_def_context(optee->teedev);
>>         if (IS_ERR(ctx)) {
>>                 rc = PTR_ERR(ctx);
>>                 goto err_rhashtable_free;
>> diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
>> index e9456e3e74cc..c77a3e631d04 100644
>> --- a/drivers/tee/optee/smc_abi.c
>> +++ b/drivers/tee/optee/smc_abi.c
>> @@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
>>         mutex_init(&optee->rpmb_dev_mutex);
>>
>>         platform_set_drvdata(pdev, optee);
>> -       ctx = teedev_open(optee->teedev);
>> +       ctx = teedev_get_def_context(optee->teedev);
>>         if (IS_ERR(ctx)) {
>>                 rc = PTR_ERR(ctx);
>>                 goto err_supp_uninit;
>> diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
>> index 93f3b330aec8..805e1336089d 100644
>> --- a/drivers/tee/tee_core.c
>> +++ b/drivers/tee/tee_core.c
>> @@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
>>                 goto err;
>>         }
>>
>> -       kref_init(&ctx->refcount);
>>         ctx->teedev = teedev;
>>         INIT_LIST_HEAD(&ctx->list_shm);
>>         rc = teedev->desc->ops->open(ctx);
>> @@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
>>  }
>>  EXPORT_SYMBOL_GPL(teedev_open);
>>
>> -void teedev_ctx_get(struct tee_context *ctx)
>> +struct tee_context *teedev_get_def_context(struct tee_device *teedev)
>>  {
>> -       if (ctx->releasing)
>> -               return;
>> +       int rc;
>> +       struct tee_context *ctx = &teedev->def_ctx;
>>
>> -       kref_get(&ctx->refcount);
>> -}
>> +       ctx->teedev = teedev;
>> +       INIT_LIST_HEAD(&ctx->list_shm);
>> +       rc = teedev->desc->ops->open(ctx);
>> +       if (rc)
>> +               return ERR_PTR(rc);
> 
> I think ctx->teedev and ctx->list_shm must always be initialized or
> &teedev->def_ctx can't be used in teedev_close_context().

True, but &teedev->def_ctx is never used in teedev_close_context().
The closing of the &teedev->def_ctx simply ignored. So once opened,
&teedev->def_ctx will always remain open until the tee_device is alive. 

> We could initialize teedev->def_ctx on the first call to teedev_open()
> on that tee_device. We need a way to tell the
> teedev->desc->ops->open() to the backed driver that it's initializing
> the default context though, or optee_open() can't handle the
> tee-supplicant case properly.
> 

That's a good point. This way, it is guaranteed that there is one def_ctx
per teedev. There should be a way to tell the open() callback that it is
a def_ctx, so it is not registered as a supplicant context.


> Should we allow this function to be called more than once for each teedev?

Yes, moving to teedev_open() will fix the issue.

> Do we need serialization in this function if it's called after the
> driver is probed?
> 

True. I'll make sure there is no race.

>>
>> -static void teedev_ctx_release(struct kref *ref)
>> -{
>> -       struct tee_context *ctx = container_of(ref, struct tee_context,
>> -                                              refcount);
>> -       ctx->releasing = true;
>> -       ctx->teedev->desc->ops->release(ctx);
>> -       kfree(ctx);
>> +       return ctx;
>>  }
>> +EXPORT_SYMBOL_GPL(teedev_get_def_context);
>>
>> -void teedev_ctx_put(struct tee_context *ctx)
>> +void teedev_close_context(struct tee_context *ctx)
>>  {
>> -       if (ctx->releasing)
>> +       struct tee_device *teedev = ctx->teedev;
>> +       struct tee_shm *shm;
>> +
>> +       if (ctx == &teedev->def_ctx)
>>                 return;
>>
>> -       kref_put(&ctx->refcount, teedev_ctx_release);
>> -}
>> +       teedev->desc->ops->release(ctx);
>>
>> -void teedev_close_context(struct tee_context *ctx)
>> -{
>> -       struct tee_device *teedev = ctx->teedev;
>> +       mutex_lock(&teedev->mutex);
>> +       list_for_each_entry(shm, &ctx->list_shm, link) {
>> +               /* Context released. However, shm still holding a teedev reference.
>> +                * Replace shm->ctx with the default context so that tee_shm_get_from_id()
>> +                * fails (i.e. it is not accessible from userspace) but shm still
>> +                * holds a valid context for further clean up, e.g. shm_unregister().
>> +                */
> 
> /*
>  * Please format
>  * multiline comments
>  * like this. Please
>  * keep the lines at
>  * max 80 columns
>  * here and at other
>  * places in the patch-
>  * set.
>  */
> 

Ack.

>> +               shm->ctx = &teedev->def_ctx;
> 
> shm->ctx will always point to a valid context, even if it is the
> default context. It seems that we can always get hold of the correct
> teedev via shm->ctx->teedev. Do we need "tee: revert removal of
> redundant teedev in struct tee_shm"?
> 

It was there in case we wanted to use NULL, but with def_ctx, it is not
necessary. I am withdrawing that commit. :).

> Shouldn't the shm be removed from the ctx->list_shm and be moved to
> teedev->def_ctx.list_shm?
>

Not really. If we put shm in the teedev->def_ctx.list_shm, by the time
we are closing the def_ctx, the list is guaranteed to be empty.

However, I understand it is cleaner and more consistent to do that rather
than making changes to tee_shm_put().

I'll do it.

>> +       }
>> +       mutex_unlock(&teedev->mutex);
>>
>> -       teedev_ctx_put(ctx);
>> +       kfree(ctx);
>>         tee_device_put(teedev);
>>  }
>>  EXPORT_SYMBOL_GPL(teedev_close_context);
>> @@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
>>
>>         teedev->desc = teedesc;
>>         teedev->pool = pool;
>> +       /* Only open default context when teedev_get_def_context() called. */
>> +       teedev->def_ctx.teedev = NULL;
>>
>>         return teedev;
>>  err_devt:
>> @@ -1027,16 +1035,31 @@ EXPORT_SYMBOL_GPL(tee_device_register);
>>
>>  void tee_device_put(struct tee_device *teedev)
>>  {
>> -       mutex_lock(&teedev->mutex);
>> -       /* Shouldn't put in this state */
>> -       if (!WARN_ON(!teedev->desc)) {
>> -               teedev->num_users--;
>> -               if (!teedev->num_users) {
>> -                       teedev->desc = NULL;
>> -                       complete(&teedev->c_no_users);
>> -               }
>> +       const struct tee_desc *desc;
>> +
>> +       scoped_guard(mutex, &teedev->mutex) {
>> +               desc = teedev->desc;
>> +
>> +               /* Shouldn't put in this state */
>> +               if (WARN_ON(!desc))
>> +                       return;
>> +
>> +               /* If there is still users for teedev */
>> +               if (--teedev->num_users)
> 
> Please do teedev->num_users-- first and then check. It makes the code
> easier to read.

Ack.

> 
>> +                       return;
>> +
>> +               /* tee_device_unregister() has been called and there is no
>> +                * user in userspace or kernel, including orphan shm for teedev.
>> +                * Set teedev->desc to NULL, so that teedev can not be reused.
>> +                */
>> +               teedev->desc = NULL;
>>         }
>> -       mutex_unlock(&teedev->mutex);
>> +
>> +       /* Release the default context */
>> +       desc->ops->release(&teedev->def_ctx);
> 
> This should only be done if teedev->def_ctx has been initialized.
> 

Ack.

> Cheers,
> Jens

Thank you very much for your comments :).
If you're okay with introducing def_ctx, I'll prepare a complete patchset
with all the details. 

Best Regards,
Amir


> 
>> +       teedev->def_ctx.teedev = NULL;
>> +
>> +       complete(&teedev->c_no_users);
>>  }
>>
>>  bool tee_device_get(struct tee_device *teedev)
>> diff --git a/drivers/tee/tee_private.h b/drivers/tee/tee_private.h
>> index 9bc50605227c..6c7bcc308958 100644
>> --- a/drivers/tee/tee_private.h
>> +++ b/drivers/tee/tee_private.h
>> @@ -17,9 +17,6 @@ int tee_shm_get_fd(struct tee_shm *shm);
>>  bool tee_device_get(struct tee_device *teedev);
>>  void tee_device_put(struct tee_device *teedev);
>>
>> -void teedev_ctx_get(struct tee_context *ctx);
>> -void teedev_ctx_put(struct tee_context *ctx);
>> -
>>  struct tee_shm *tee_shm_alloc_user_buf(struct tee_context *ctx, size_t size);
>>  struct tee_shm *tee_shm_register_user_buf(struct tee_context *ctx,
>>                                           unsigned long addr, size_t length);
>> diff --git a/drivers/tee/tee_shm.c b/drivers/tee/tee_shm.c
>> index c0164c0f4a01..f07274291edf 100644
>> --- a/drivers/tee/tee_shm.c
>> +++ b/drivers/tee/tee_shm.c
>> @@ -59,8 +59,6 @@ static void tee_shm_release(struct tee_shm *shm)
>>                 release_registered_pages(shm);
>>         }
>>
>> -       teedev_ctx_put(shm->ctx);
>> -
>>         kfree(shm);
>>
>>         tee_device_put(teedev);
>> @@ -93,13 +91,6 @@ static struct tee_shm *shm_alloc_helper(struct tee_context *ctx, size_t size,
>>         shm->flags = flags;
>>         shm->teedev = teedev;
>>         shm->id = id;
>> -
>> -       /*
>> -        * We're assigning this as it is needed if the shm is to be
>> -        * registered. If this function returns OK then the caller expected
>> -        * to call teedev_ctx_get() or clear shm->ctx in case it's not
>> -        * needed any longer.
>> -        */
>>         shm->ctx = ctx;
>>
>>         rc = teedev->pool->ops->alloc(teedev->pool, shm, size, align);
>> @@ -112,7 +103,6 @@ static struct tee_shm *shm_alloc_helper(struct tee_context *ctx, size_t size,
>>         list_add_tail(&shm->link, &ctx->list_shm);
>>         mutex_unlock(&teedev->mutex);
>>
>> -       teedev_ctx_get(ctx);
>>         return shm;
>>  err_kfree:
>>         kfree(shm);
>> @@ -295,12 +285,10 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
>>                 goto err_dev_put;
>>         }
>>
>> -       teedev_ctx_get(ctx);
>> -
>>         shm = kzalloc(sizeof(*shm), GFP_KERNEL);
>>         if (!shm) {
>>                 ret = ERR_PTR(-ENOMEM);
>> -               goto err_ctx_put;
>> +               goto err_dev_put;
>>         }
>>
>>         refcount_set(&shm->refcount, 1);
>> @@ -313,7 +301,7 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
>>         num_pages = iov_iter_npages(iter, INT_MAX);
>>         if (!num_pages) {
>>                 ret = ERR_PTR(-ENOMEM);
>> -               goto err_ctx_put;
>> +               goto err_dev_put;
>>         }
>>
>>         shm->pages = kcalloc(num_pages, sizeof(*shm->pages), GFP_KERNEL);
>> @@ -361,8 +349,6 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
>>         kfree(shm->pages);
>>  err_free_shm:
>>         kfree(shm);
>> -err_ctx_put:
>> -       teedev_ctx_put(ctx);
>>  err_dev_put:
>>         tee_device_put(teedev);
>>         return ret;
>> diff --git a/include/linux/tee_core.h b/include/linux/tee_core.h
>> index a38494d6b5f4..13393ddac530 100644
>> --- a/include/linux/tee_core.h
>> +++ b/include/linux/tee_core.h
>> @@ -44,6 +44,7 @@
>>   * @idr:       register of user space shared memory objects allocated or
>>   *             registered on this device
>>   * @pool:      shared memory pool
>> + * @def_ctx:   default context used if there is no context available, e.g. internal driver calls.
>>   */
>>  struct tee_device {
>>         char name[TEE_MAX_DEV_NAME_LEN];
>> @@ -60,6 +61,7 @@ struct tee_device {
>>
>>         struct idr idr;
>>         struct tee_shm_pool *pool;
>> +       struct tee_context def_ctx;
>>  };
>>
>>  /**
>> @@ -309,6 +311,19 @@ static inline bool tee_param_is_memref(struct tee_param *param)
>>   */
>>  struct tee_context *teedev_open(struct tee_device *teedev);
>>
>> +/**
>> + * teedev_get_def_context() - Get default context for a struct tee_device
>> + * @teedev:    Device to open
>> + *
>> + * Unlike a context that returned from teedev_open(), the default context is static
>> + * and available as long as @teedev has a user ''other then this context''. This context
>> + * can be used for driver internal operation and clean up where a context should be
>> + * available, while tee_device_unregister() is waiting for other users to go away.
>> + *
>> + * @return a pointer to struct tee_context on success or an ERR_PTR on failure.
>> + */
>> +struct tee_context *teedev_get_def_context(struct tee_device *teedev);
>> +
>>  /**
>>   * teedev_close_context() - closes a struct tee_context
>>   * @ctx:       The struct tee_context to close
>> diff --git a/include/linux/tee_drv.h b/include/linux/tee_drv.h
>> index 1b57cddfecc8..9633e14ba484 100644
>> --- a/include/linux/tee_drv.h
>> +++ b/include/linux/tee_drv.h
>> @@ -7,7 +7,6 @@
>>  #define __TEE_DRV_H
>>
>>  #include <linux/device.h>
>> -#include <linux/kref.h>
>>  #include <linux/list.h>
>>  #include <linux/mod_devicetable.h>
>>  #include <linux/tee.h>
>> @@ -25,10 +24,6 @@ struct tee_device;
>>   * @teedev:    pointer to this drivers struct tee_device
>>   * @list_shm:  List of shared memory object owned by this context
>>   * @data:      driver specific context data, managed by the driver
>> - * @refcount:  reference counter for this structure
>> - * @releasing:  flag that indicates if context is being released right now.
>> - *             It is needed to break circular dependency on context during
>> - *              shared memory release.
>>   * @supp_nowait: flag that indicates that requests in this context should not
>>   *              wait for tee-supplicant daemon to be started if not present
>>   *              and just return with an error code. It is needed for requests
>> @@ -41,8 +36,6 @@ struct tee_context {
>>         struct tee_device *teedev;
>>         struct list_head list_shm;
>>         void *data;
>> -       struct kref refcount;
>> -       bool releasing;
>>         bool supp_nowait;
>>         bool cap_memref_null;
>>  };
>>
>> --
>> 2.34.1
>>

Re: [PATCH RFC 3/3] tee: introduce orphan tee_shm and default context

[Sumit Garg]

Hi Amirreza,

Thanks for proposing this.

On Fri, 22 Nov 2024 at 06:38, Amirreza Zarrabi
<quic_azarrabi@quicinc.com> wrote:
>
>
> On 11/21/2024 11:08 PM, Jens Wiklander wrote:
>
> Hi Jens,
>
> > Hi Amirreza,
> >
> > On Thu, Nov 21, 2024 at 2:37 AM Amirreza Zarrabi
> > <quic_azarrabi@quicinc.com> wrote:
> >>
> >> The default context has a lifespan similar to the tee_device.

Since it's associated with tee_device context, let's call it obvious
via renaming it as device context instead (s/def_ctx/dev_ctx/ in this
patch).

> >> It is used as a context for shared memory if the context to which the
> >> shared memory belongs is released, making the tee_shm an orphan.
> >> This allows the driver implementing shm_unregister to safely make
> >> subsequent calls, such as to a supplicant if needed.
> >>
> >> It also enables users to free the shared memory while the driver is
> >> blocked on unregister_tee_device safely.
> >>
> >> Preferably, this should be used for all driver internal uses, using
> >> teedev_get_def_context rather than calling teedev_open.

Makes sense to me.

> >>
> >> Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
> >> ---
> >>  drivers/tee/optee/core.c    |  2 +-
> >>  drivers/tee/optee/ffa_abi.c |  2 +-
> >>  drivers/tee/optee/smc_abi.c |  2 +-
> >>  drivers/tee/tee_core.c      | 83 +++++++++++++++++++++++++++++----------------
> >>  drivers/tee/tee_private.h   |  3 --
> >>  drivers/tee/tee_shm.c       | 18 ++--------
> >>  include/linux/tee_core.h    | 15 ++++++++
> >>  include/linux/tee_drv.h     |  7 ----
> >>  8 files changed, 73 insertions(+), 59 deletions(-)
> >>
> >> diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
> >> index c75fddc83576..78d43d0c8014 100644
> >> --- a/drivers/tee/optee/core.c
> >> +++ b/drivers/tee/optee/core.c
> >> @@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
> >>
> >>         optee_notif_uninit(optee);
> >>         optee_shm_arg_cache_uninit(optee);
> >> -       teedev_close_context(optee->ctx);
> >> +
> >>         /*
> >>          * The two devices have to be unregistered before we can free the
> >>          * other resources.
> >> diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
> >> index f3af5666bb11..6ad94f0788ad 100644
> >> --- a/drivers/tee/optee/ffa_abi.c
> >> +++ b/drivers/tee/optee/ffa_abi.c
> >> @@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
> >>         optee_shm_arg_cache_init(optee, arg_cache_flags);
> >>         mutex_init(&optee->rpmb_dev_mutex);
> >>         ffa_dev_set_drvdata(ffa_dev, optee);
> >> -       ctx = teedev_open(optee->teedev);
> >> +       ctx = teedev_get_def_context(optee->teedev);
> >>         if (IS_ERR(ctx)) {
> >>                 rc = PTR_ERR(ctx);
> >>                 goto err_rhashtable_free;
> >> diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
> >> index e9456e3e74cc..c77a3e631d04 100644
> >> --- a/drivers/tee/optee/smc_abi.c
> >> +++ b/drivers/tee/optee/smc_abi.c
> >> @@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
> >>         mutex_init(&optee->rpmb_dev_mutex);
> >>
> >>         platform_set_drvdata(pdev, optee);
> >> -       ctx = teedev_open(optee->teedev);
> >> +       ctx = teedev_get_def_context(optee->teedev);
> >>         if (IS_ERR(ctx)) {
> >>                 rc = PTR_ERR(ctx);
> >>                 goto err_supp_uninit;
> >> diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
> >> index 93f3b330aec8..805e1336089d 100644
> >> --- a/drivers/tee/tee_core.c
> >> +++ b/drivers/tee/tee_core.c
> >> @@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> >>                 goto err;
> >>         }
> >>
> >> -       kref_init(&ctx->refcount);
> >>         ctx->teedev = teedev;
> >>         INIT_LIST_HEAD(&ctx->list_shm);
> >>         rc = teedev->desc->ops->open(ctx);
> >> @@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> >>  }
> >>  EXPORT_SYMBOL_GPL(teedev_open);
> >>
> >> -void teedev_ctx_get(struct tee_context *ctx)
> >> +struct tee_context *teedev_get_def_context(struct tee_device *teedev)
> >>  {
> >> -       if (ctx->releasing)
> >> -               return;
> >> +       int rc;
> >> +       struct tee_context *ctx = &teedev->def_ctx;
> >>
> >> -       kref_get(&ctx->refcount);
> >> -}
> >> +       ctx->teedev = teedev;
> >> +       INIT_LIST_HEAD(&ctx->list_shm);
> >> +       rc = teedev->desc->ops->open(ctx);
> >> +       if (rc)
> >> +               return ERR_PTR(rc);
> >
> > I think ctx->teedev and ctx->list_shm must always be initialized or
> > &teedev->def_ctx can't be used in teedev_close_context().
>
> True, but &teedev->def_ctx is never used in teedev_close_context().
> The closing of the &teedev->def_ctx simply ignored. So once opened,
> &teedev->def_ctx will always remain open until the tee_device is alive.
>
> > We could initialize teedev->def_ctx on the first call to teedev_open()
> > on that tee_device. We need a way to tell the
> > teedev->desc->ops->open() to the backed driver that it's initializing
> > the default context though, or optee_open() can't handle the
> > tee-supplicant case properly.
> >
>
> That's a good point. This way, it is guaranteed that there is one def_ctx
> per teedev. There should be a way to tell the open() callback that it is
> a def_ctx, so it is not registered as a supplicant context.
>
>
> > Should we allow this function to be called more than once for each teedev?
>
> Yes, moving to teedev_open() will fix the issue.
>
> > Do we need serialization in this function if it's called after the
> > driver is probed?
> >
>
> True. I'll make sure there is no race.
>
> >>
> >> -static void teedev_ctx_release(struct kref *ref)
> >> -{
> >> -       struct tee_context *ctx = container_of(ref, struct tee_context,
> >> -                                              refcount);
> >> -       ctx->releasing = true;
> >> -       ctx->teedev->desc->ops->release(ctx);
> >> -       kfree(ctx);
> >> +       return ctx;
> >>  }
> >> +EXPORT_SYMBOL_GPL(teedev_get_def_context);
> >>
> >> -void teedev_ctx_put(struct tee_context *ctx)
> >> +void teedev_close_context(struct tee_context *ctx)
> >>  {
> >> -       if (ctx->releasing)
> >> +       struct tee_device *teedev = ctx->teedev;
> >> +       struct tee_shm *shm;
> >> +
> >> +       if (ctx == &teedev->def_ctx)
> >>                 return;
> >>
> >> -       kref_put(&ctx->refcount, teedev_ctx_release);
> >> -}
> >> +       teedev->desc->ops->release(ctx);
> >>
> >> -void teedev_close_context(struct tee_context *ctx)
> >> -{
> >> -       struct tee_device *teedev = ctx->teedev;
> >> +       mutex_lock(&teedev->mutex);
> >> +       list_for_each_entry(shm, &ctx->list_shm, link) {
> >> +               /* Context released. However, shm still holding a teedev reference.
> >> +                * Replace shm->ctx with the default context so that tee_shm_get_from_id()
> >> +                * fails (i.e. it is not accessible from userspace) but shm still
> >> +                * holds a valid context for further clean up, e.g. shm_unregister().
> >> +                */
> >
> > /*
> >  * Please format
> >  * multiline comments
> >  * like this. Please
> >  * keep the lines at
> >  * max 80 columns
> >  * here and at other
> >  * places in the patch-
> >  * set.
> >  */
> >
>
> Ack.
>
> >> +               shm->ctx = &teedev->def_ctx;
> >
> > shm->ctx will always point to a valid context, even if it is the
> > default context. It seems that we can always get hold of the correct
> > teedev via shm->ctx->teedev. Do we need "tee: revert removal of
> > redundant teedev in struct tee_shm"?
> >
>
> It was there in case we wanted to use NULL, but with def_ctx, it is not
> necessary. I am withdrawing that commit. :).
>
> > Shouldn't the shm be removed from the ctx->list_shm and be moved to
> > teedev->def_ctx.list_shm?

+1

> >
>
> Not really. If we put shm in the teedev->def_ctx.list_shm, by the time
> we are closing the def_ctx, the list is guaranteed to be empty.
>
> However, I understand it is cleaner and more consistent to do that rather
> than making changes to tee_shm_put().
>
> I'll do it.
>
> >> +       }
> >> +       mutex_unlock(&teedev->mutex);
> >>
> >> -       teedev_ctx_put(ctx);
> >> +       kfree(ctx);
> >>         tee_device_put(teedev);
> >>  }
> >>  EXPORT_SYMBOL_GPL(teedev_close_context);
> >> @@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
> >>
> >>         teedev->desc = teedesc;
> >>         teedev->pool = pool;
> >> +       /* Only open default context when teedev_get_def_context() called. */
> >> +       teedev->def_ctx.teedev = NULL;

Why don't you open the device context here only? This will associate
it automatically with teedev lifespan and then
teedev_get_def_context() will just return a reference to that.

-Sumit

> >>
> >>         return teedev;
> >>  err_devt:
> >> @@ -1027,16 +1035,31 @@ EXPORT_SYMBOL_GPL(tee_device_register);
> >>
> >>  void tee_device_put(struct tee_device *teedev)
> >>  {
> >> -       mutex_lock(&teedev->mutex);
> >> -       /* Shouldn't put in this state */
> >> -       if (!WARN_ON(!teedev->desc)) {
> >> -               teedev->num_users--;
> >> -               if (!teedev->num_users) {
> >> -                       teedev->desc = NULL;
> >> -                       complete(&teedev->c_no_users);
> >> -               }
> >> +       const struct tee_desc *desc;
> >> +
> >> +       scoped_guard(mutex, &teedev->mutex) {
> >> +               desc = teedev->desc;
> >> +
> >> +               /* Shouldn't put in this state */
> >> +               if (WARN_ON(!desc))
> >> +                       return;
> >> +
> >> +               /* If there is still users for teedev */
> >> +               if (--teedev->num_users)
> >
> > Please do teedev->num_users-- first and then check. It makes the code
> > easier to read.
>
> Ack.
>
> >
> >> +                       return;
> >> +
> >> +               /* tee_device_unregister() has been called and there is no
> >> +                * user in userspace or kernel, including orphan shm for teedev.
> >> +                * Set teedev->desc to NULL, so that teedev can not be reused.
> >> +                */
> >> +               teedev->desc = NULL;
> >>         }
> >> -       mutex_unlock(&teedev->mutex);
> >> +
> >> +       /* Release the default context */
> >> +       desc->ops->release(&teedev->def_ctx);
> >
> > This should only be done if teedev->def_ctx has been initialized.
> >
>
> Ack.
>
> > Cheers,
> > Jens
>
> Thank you very much for your comments :).
> If you're okay with introducing def_ctx, I'll prepare a complete patchset
> with all the details.
>
> Best Regards,
> Amir
>
>
> >
> >> +       teedev->def_ctx.teedev = NULL;
> >> +
> >> +       complete(&teedev->c_no_users);
> >>  }
> >>
> >>  bool tee_device_get(struct tee_device *teedev)
> >> diff --git a/drivers/tee/tee_private.h b/drivers/tee/tee_private.h
> >> index 9bc50605227c..6c7bcc308958 100644
> >> --- a/drivers/tee/tee_private.h
> >> +++ b/drivers/tee/tee_private.h
> >> @@ -17,9 +17,6 @@ int tee_shm_get_fd(struct tee_shm *shm);
> >>  bool tee_device_get(struct tee_device *teedev);
> >>  void tee_device_put(struct tee_device *teedev);
> >>
> >> -void teedev_ctx_get(struct tee_context *ctx);
> >> -void teedev_ctx_put(struct tee_context *ctx);
> >> -
> >>  struct tee_shm *tee_shm_alloc_user_buf(struct tee_context *ctx, size_t size);
> >>  struct tee_shm *tee_shm_register_user_buf(struct tee_context *ctx,
> >>                                           unsigned long addr, size_t length);
> >> diff --git a/drivers/tee/tee_shm.c b/drivers/tee/tee_shm.c
> >> index c0164c0f4a01..f07274291edf 100644
> >> --- a/drivers/tee/tee_shm.c
> >> +++ b/drivers/tee/tee_shm.c
> >> @@ -59,8 +59,6 @@ static void tee_shm_release(struct tee_shm *shm)
> >>                 release_registered_pages(shm);
> >>         }
> >>
> >> -       teedev_ctx_put(shm->ctx);
> >> -
> >>         kfree(shm);
> >>
> >>         tee_device_put(teedev);
> >> @@ -93,13 +91,6 @@ static struct tee_shm *shm_alloc_helper(struct tee_context *ctx, size_t size,
> >>         shm->flags = flags;
> >>         shm->teedev = teedev;
> >>         shm->id = id;
> >> -
> >> -       /*
> >> -        * We're assigning this as it is needed if the shm is to be
> >> -        * registered. If this function returns OK then the caller expected
> >> -        * to call teedev_ctx_get() or clear shm->ctx in case it's not
> >> -        * needed any longer.
> >> -        */
> >>         shm->ctx = ctx;
> >>
> >>         rc = teedev->pool->ops->alloc(teedev->pool, shm, size, align);
> >> @@ -112,7 +103,6 @@ static struct tee_shm *shm_alloc_helper(struct tee_context *ctx, size_t size,
> >>         list_add_tail(&shm->link, &ctx->list_shm);
> >>         mutex_unlock(&teedev->mutex);
> >>
> >> -       teedev_ctx_get(ctx);
> >>         return shm;
> >>  err_kfree:
> >>         kfree(shm);
> >> @@ -295,12 +285,10 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
> >>                 goto err_dev_put;
> >>         }
> >>
> >> -       teedev_ctx_get(ctx);
> >> -
> >>         shm = kzalloc(sizeof(*shm), GFP_KERNEL);
> >>         if (!shm) {
> >>                 ret = ERR_PTR(-ENOMEM);
> >> -               goto err_ctx_put;
> >> +               goto err_dev_put;
> >>         }
> >>
> >>         refcount_set(&shm->refcount, 1);
> >> @@ -313,7 +301,7 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
> >>         num_pages = iov_iter_npages(iter, INT_MAX);
> >>         if (!num_pages) {
> >>                 ret = ERR_PTR(-ENOMEM);
> >> -               goto err_ctx_put;
> >> +               goto err_dev_put;
> >>         }
> >>
> >>         shm->pages = kcalloc(num_pages, sizeof(*shm->pages), GFP_KERNEL);
> >> @@ -361,8 +349,6 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
> >>         kfree(shm->pages);
> >>  err_free_shm:
> >>         kfree(shm);
> >> -err_ctx_put:
> >> -       teedev_ctx_put(ctx);
> >>  err_dev_put:
> >>         tee_device_put(teedev);
> >>         return ret;
> >> diff --git a/include/linux/tee_core.h b/include/linux/tee_core.h
> >> index a38494d6b5f4..13393ddac530 100644
> >> --- a/include/linux/tee_core.h
> >> +++ b/include/linux/tee_core.h
> >> @@ -44,6 +44,7 @@
> >>   * @idr:       register of user space shared memory objects allocated or
> >>   *             registered on this device
> >>   * @pool:      shared memory pool
> >> + * @def_ctx:   default context used if there is no context available, e.g. internal driver calls.
> >>   */
> >>  struct tee_device {
> >>         char name[TEE_MAX_DEV_NAME_LEN];
> >> @@ -60,6 +61,7 @@ struct tee_device {
> >>
> >>         struct idr idr;
> >>         struct tee_shm_pool *pool;
> >> +       struct tee_context def_ctx;
> >>  };
> >>
> >>  /**
> >> @@ -309,6 +311,19 @@ static inline bool tee_param_is_memref(struct tee_param *param)
> >>   */
> >>  struct tee_context *teedev_open(struct tee_device *teedev);
> >>
> >> +/**
> >> + * teedev_get_def_context() - Get default context for a struct tee_device
> >> + * @teedev:    Device to open
> >> + *
> >> + * Unlike a context that returned from teedev_open(), the default context is static
> >> + * and available as long as @teedev has a user ''other then this context''. This context
> >> + * can be used for driver internal operation and clean up where a context should be
> >> + * available, while tee_device_unregister() is waiting for other users to go away.
> >> + *
> >> + * @return a pointer to struct tee_context on success or an ERR_PTR on failure.
> >> + */
> >> +struct tee_context *teedev_get_def_context(struct tee_device *teedev);
> >> +
> >>  /**
> >>   * teedev_close_context() - closes a struct tee_context
> >>   * @ctx:       The struct tee_context to close
> >> diff --git a/include/linux/tee_drv.h b/include/linux/tee_drv.h
> >> index 1b57cddfecc8..9633e14ba484 100644
> >> --- a/include/linux/tee_drv.h
> >> +++ b/include/linux/tee_drv.h
> >> @@ -7,7 +7,6 @@
> >>  #define __TEE_DRV_H
> >>
> >>  #include <linux/device.h>
> >> -#include <linux/kref.h>
> >>  #include <linux/list.h>
> >>  #include <linux/mod_devicetable.h>
> >>  #include <linux/tee.h>
> >> @@ -25,10 +24,6 @@ struct tee_device;
> >>   * @teedev:    pointer to this drivers struct tee_device
> >>   * @list_shm:  List of shared memory object owned by this context
> >>   * @data:      driver specific context data, managed by the driver
> >> - * @refcount:  reference counter for this structure
> >> - * @releasing:  flag that indicates if context is being released right now.
> >> - *             It is needed to break circular dependency on context during
> >> - *              shared memory release.
> >>   * @supp_nowait: flag that indicates that requests in this context should not
> >>   *              wait for tee-supplicant daemon to be started if not present
> >>   *              and just return with an error code. It is needed for requests
> >> @@ -41,8 +36,6 @@ struct tee_context {
> >>         struct tee_device *teedev;
> >>         struct list_head list_shm;
> >>         void *data;
> >> -       struct kref refcount;
> >> -       bool releasing;
> >>         bool supp_nowait;
> >>         bool cap_memref_null;
> >>  };
> >>
> >> --
> >> 2.34.1
> >>

Re: [PATCH RFC 3/3] tee: introduce orphan tee_shm and default context

[Amirreza Zarrabi]

Hi Sumit,

Thank you so much for the comemnts :).

On 11/23/2024 9:32 PM, Sumit Garg wrote:
> Hi Amirreza,
> 
> Thanks for proposing this.
> 
> On Fri, 22 Nov 2024 at 06:38, Amirreza Zarrabi
> <quic_azarrabi@quicinc.com> wrote:
>>
>>
>> On 11/21/2024 11:08 PM, Jens Wiklander wrote:
>>
>> Hi Jens,
>>
>>> Hi Amirreza,
>>>
>>> On Thu, Nov 21, 2024 at 2:37 AM Amirreza Zarrabi
>>> <quic_azarrabi@quicinc.com> wrote:
>>>>
>>>> The default context has a lifespan similar to the tee_device.
> 
> Since it's associated with tee_device context, let's call it obvious
> via renaming it as device context instead (s/def_ctx/dev_ctx/ in this
> patch).
> 

Make sense, I'll rename it.

>>>> It is used as a context for shared memory if the context to which the
>>>> shared memory belongs is released, making the tee_shm an orphan.
>>>> This allows the driver implementing shm_unregister to safely make
>>>> subsequent calls, such as to a supplicant if needed.
>>>>
>>>> It also enables users to free the shared memory while the driver is
>>>> blocked on unregister_tee_device safely.
>>>>
>>>> Preferably, this should be used for all driver internal uses, using
>>>> teedev_get_def_context rather than calling teedev_open.
> 
> Makes sense to me.
> 
>>>>
>>>> Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
>>>> ---
>>>>  drivers/tee/optee/core.c    |  2 +-
>>>>  drivers/tee/optee/ffa_abi.c |  2 +-
>>>>  drivers/tee/optee/smc_abi.c |  2 +-
>>>>  drivers/tee/tee_core.c      | 83 +++++++++++++++++++++++++++++----------------
>>>>  drivers/tee/tee_private.h   |  3 --
>>>>  drivers/tee/tee_shm.c       | 18 ++--------
>>>>  include/linux/tee_core.h    | 15 ++++++++
>>>>  include/linux/tee_drv.h     |  7 ----
>>>>  8 files changed, 73 insertions(+), 59 deletions(-)
>>>>
>>>> diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
>>>> index c75fddc83576..78d43d0c8014 100644
>>>> --- a/drivers/tee/optee/core.c
>>>> +++ b/drivers/tee/optee/core.c
>>>> @@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
>>>>
>>>>         optee_notif_uninit(optee);
>>>>         optee_shm_arg_cache_uninit(optee);
>>>> -       teedev_close_context(optee->ctx);
>>>> +
>>>>         /*
>>>>          * The two devices have to be unregistered before we can free the
>>>>          * other resources.
>>>> diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
>>>> index f3af5666bb11..6ad94f0788ad 100644
>>>> --- a/drivers/tee/optee/ffa_abi.c
>>>> +++ b/drivers/tee/optee/ffa_abi.c
>>>> @@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
>>>>         optee_shm_arg_cache_init(optee, arg_cache_flags);
>>>>         mutex_init(&optee->rpmb_dev_mutex);
>>>>         ffa_dev_set_drvdata(ffa_dev, optee);
>>>> -       ctx = teedev_open(optee->teedev);
>>>> +       ctx = teedev_get_def_context(optee->teedev);
>>>>         if (IS_ERR(ctx)) {
>>>>                 rc = PTR_ERR(ctx);
>>>>                 goto err_rhashtable_free;
>>>> diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
>>>> index e9456e3e74cc..c77a3e631d04 100644
>>>> --- a/drivers/tee/optee/smc_abi.c
>>>> +++ b/drivers/tee/optee/smc_abi.c
>>>> @@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
>>>>         mutex_init(&optee->rpmb_dev_mutex);
>>>>
>>>>         platform_set_drvdata(pdev, optee);
>>>> -       ctx = teedev_open(optee->teedev);
>>>> +       ctx = teedev_get_def_context(optee->teedev);
>>>>         if (IS_ERR(ctx)) {
>>>>                 rc = PTR_ERR(ctx);
>>>>                 goto err_supp_uninit;
>>>> diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
>>>> index 93f3b330aec8..805e1336089d 100644
>>>> --- a/drivers/tee/tee_core.c
>>>> +++ b/drivers/tee/tee_core.c
>>>> @@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
>>>>                 goto err;
>>>>         }
>>>>
>>>> -       kref_init(&ctx->refcount);
>>>>         ctx->teedev = teedev;
>>>>         INIT_LIST_HEAD(&ctx->list_shm);
>>>>         rc = teedev->desc->ops->open(ctx);
>>>> @@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
>>>>  }
>>>>  EXPORT_SYMBOL_GPL(teedev_open);
>>>>
>>>> -void teedev_ctx_get(struct tee_context *ctx)
>>>> +struct tee_context *teedev_get_def_context(struct tee_device *teedev)
>>>>  {
>>>> -       if (ctx->releasing)
>>>> -               return;
>>>> +       int rc;
>>>> +       struct tee_context *ctx = &teedev->def_ctx;
>>>>
>>>> -       kref_get(&ctx->refcount);
>>>> -}
>>>> +       ctx->teedev = teedev;
>>>> +       INIT_LIST_HEAD(&ctx->list_shm);
>>>> +       rc = teedev->desc->ops->open(ctx);
>>>> +       if (rc)
>>>> +               return ERR_PTR(rc);
>>>
>>> I think ctx->teedev and ctx->list_shm must always be initialized or
>>> &teedev->def_ctx can't be used in teedev_close_context().
>>
>> True, but &teedev->def_ctx is never used in teedev_close_context().
>> The closing of the &teedev->def_ctx simply ignored. So once opened,
>> &teedev->def_ctx will always remain open until the tee_device is alive.
>>
>>> We could initialize teedev->def_ctx on the first call to teedev_open()
>>> on that tee_device. We need a way to tell the
>>> teedev->desc->ops->open() to the backed driver that it's initializing
>>> the default context though, or optee_open() can't handle the
>>> tee-supplicant case properly.
>>>
>>
>> That's a good point. This way, it is guaranteed that there is one def_ctx
>> per teedev. There should be a way to tell the open() callback that it is
>> a def_ctx, so it is not registered as a supplicant context.
>>
>>
>>> Should we allow this function to be called more than once for each teedev?
>>
>> Yes, moving to teedev_open() will fix the issue.
>>
>>> Do we need serialization in this function if it's called after the
>>> driver is probed?
>>>
>>
>> True. I'll make sure there is no race.
>>
>>>>
>>>> -static void teedev_ctx_release(struct kref *ref)
>>>> -{
>>>> -       struct tee_context *ctx = container_of(ref, struct tee_context,
>>>> -                                              refcount);
>>>> -       ctx->releasing = true;
>>>> -       ctx->teedev->desc->ops->release(ctx);
>>>> -       kfree(ctx);
>>>> +       return ctx;
>>>>  }
>>>> +EXPORT_SYMBOL_GPL(teedev_get_def_context);
>>>>
>>>> -void teedev_ctx_put(struct tee_context *ctx)
>>>> +void teedev_close_context(struct tee_context *ctx)
>>>>  {
>>>> -       if (ctx->releasing)
>>>> +       struct tee_device *teedev = ctx->teedev;
>>>> +       struct tee_shm *shm;
>>>> +
>>>> +       if (ctx == &teedev->def_ctx)
>>>>                 return;
>>>>
>>>> -       kref_put(&ctx->refcount, teedev_ctx_release);
>>>> -}
>>>> +       teedev->desc->ops->release(ctx);
>>>>
>>>> -void teedev_close_context(struct tee_context *ctx)
>>>> -{
>>>> -       struct tee_device *teedev = ctx->teedev;
>>>> +       mutex_lock(&teedev->mutex);
>>>> +       list_for_each_entry(shm, &ctx->list_shm, link) {
>>>> +               /* Context released. However, shm still holding a teedev reference.
>>>> +                * Replace shm->ctx with the default context so that tee_shm_get_from_id()
>>>> +                * fails (i.e. it is not accessible from userspace) but shm still
>>>> +                * holds a valid context for further clean up, e.g. shm_unregister().
>>>> +                */
>>>
>>> /*
>>>  * Please format
>>>  * multiline comments
>>>  * like this. Please
>>>  * keep the lines at
>>>  * max 80 columns
>>>  * here and at other
>>>  * places in the patch-
>>>  * set.
>>>  */
>>>
>>
>> Ack.
>>
>>>> +               shm->ctx = &teedev->def_ctx;
>>>
>>> shm->ctx will always point to a valid context, even if it is the
>>> default context. It seems that we can always get hold of the correct
>>> teedev via shm->ctx->teedev. Do we need "tee: revert removal of
>>> redundant teedev in struct tee_shm"?
>>>
>>
>> It was there in case we wanted to use NULL, but with def_ctx, it is not
>> necessary. I am withdrawing that commit. :).
>>
>>> Shouldn't the shm be removed from the ctx->list_shm and be moved to
>>> teedev->def_ctx.list_shm?
> 
> +1
> 

Ack.

>>>
>>
>> Not really. If we put shm in the teedev->def_ctx.list_shm, by the time
>> we are closing the def_ctx, the list is guaranteed to be empty.
>>
>> However, I understand it is cleaner and more consistent to do that rather
>> than making changes to tee_shm_put().
>>
>> I'll do it.
>>
>>>> +       }
>>>> +       mutex_unlock(&teedev->mutex);
>>>>
>>>> -       teedev_ctx_put(ctx);
>>>> +       kfree(ctx);
>>>>         tee_device_put(teedev);
>>>>  }
>>>>  EXPORT_SYMBOL_GPL(teedev_close_context);
>>>> @@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
>>>>
>>>>         teedev->desc = teedesc;
>>>>         teedev->pool = pool;
>>>> +       /* Only open default context when teedev_get_def_context() called. */
>>>> +       teedev->def_ctx.teedev = NULL;
> 
> Why don't you open the device context here only? This will associate
> it automatically with teedev lifespan and then
> teedev_get_def_context() will just return a reference to that.
> 
> -Sumit
> 

So my assumption is that the tee_devic_alloc() is called as part of
the driver initialization; there is no guarantee that at this time the
driver is actually ready to accept any open() callback.

Best Regards,
Amir

>>>>
>>>>         return teedev;
>>>>  err_devt:
>>>> @@ -1027,16 +1035,31 @@ EXPORT_SYMBOL_GPL(tee_device_register);
>>>>
>>>>  void tee_device_put(struct tee_device *teedev)
>>>>  {
>>>> -       mutex_lock(&teedev->mutex);
>>>> -       /* Shouldn't put in this state */
>>>> -       if (!WARN_ON(!teedev->desc)) {
>>>> -               teedev->num_users--;
>>>> -               if (!teedev->num_users) {
>>>> -                       teedev->desc = NULL;
>>>> -                       complete(&teedev->c_no_users);
>>>> -               }
>>>> +       const struct tee_desc *desc;
>>>> +
>>>> +       scoped_guard(mutex, &teedev->mutex) {
>>>> +               desc = teedev->desc;
>>>> +
>>>> +               /* Shouldn't put in this state */
>>>> +               if (WARN_ON(!desc))
>>>> +                       return;
>>>> +
>>>> +               /* If there is still users for teedev */
>>>> +               if (--teedev->num_users)
>>>
>>> Please do teedev->num_users-- first and then check. It makes the code
>>> easier to read.
>>
>> Ack.
>>
>>>
>>>> +                       return;
>>>> +
>>>> +               /* tee_device_unregister() has been called and there is no
>>>> +                * user in userspace or kernel, including orphan shm for teedev.
>>>> +                * Set teedev->desc to NULL, so that teedev can not be reused.
>>>> +                */
>>>> +               teedev->desc = NULL;
>>>>         }
>>>> -       mutex_unlock(&teedev->mutex);
>>>> +
>>>> +       /* Release the default context */
>>>> +       desc->ops->release(&teedev->def_ctx);
>>>
>>> This should only be done if teedev->def_ctx has been initialized.
>>>
>>
>> Ack.
>>
>>> Cheers,
>>> Jens
>>
>> Thank you very much for your comments :).
>> If you're okay with introducing def_ctx, I'll prepare a complete patchset
>> with all the details.
>>
>> Best Regards,
>> Amir
>>
>>
>>>
>>>> +       teedev->def_ctx.teedev = NULL;
>>>> +
>>>> +       complete(&teedev->c_no_users);
>>>>  }
>>>>
>>>>  bool tee_device_get(struct tee_device *teedev)
>>>> diff --git a/drivers/tee/tee_private.h b/drivers/tee/tee_private.h
>>>> index 9bc50605227c..6c7bcc308958 100644
>>>> --- a/drivers/tee/tee_private.h
>>>> +++ b/drivers/tee/tee_private.h
>>>> @@ -17,9 +17,6 @@ int tee_shm_get_fd(struct tee_shm *shm);
>>>>  bool tee_device_get(struct tee_device *teedev);
>>>>  void tee_device_put(struct tee_device *teedev);
>>>>
>>>> -void teedev_ctx_get(struct tee_context *ctx);
>>>> -void teedev_ctx_put(struct tee_context *ctx);
>>>> -
>>>>  struct tee_shm *tee_shm_alloc_user_buf(struct tee_context *ctx, size_t size);
>>>>  struct tee_shm *tee_shm_register_user_buf(struct tee_context *ctx,
>>>>                                           unsigned long addr, size_t length);
>>>> diff --git a/drivers/tee/tee_shm.c b/drivers/tee/tee_shm.c
>>>> index c0164c0f4a01..f07274291edf 100644
>>>> --- a/drivers/tee/tee_shm.c
>>>> +++ b/drivers/tee/tee_shm.c
>>>> @@ -59,8 +59,6 @@ static void tee_shm_release(struct tee_shm *shm)
>>>>                 release_registered_pages(shm);
>>>>         }
>>>>
>>>> -       teedev_ctx_put(shm->ctx);
>>>> -
>>>>         kfree(shm);
>>>>
>>>>         tee_device_put(teedev);
>>>> @@ -93,13 +91,6 @@ static struct tee_shm *shm_alloc_helper(struct tee_context *ctx, size_t size,
>>>>         shm->flags = flags;
>>>>         shm->teedev = teedev;
>>>>         shm->id = id;
>>>> -
>>>> -       /*
>>>> -        * We're assigning this as it is needed if the shm is to be
>>>> -        * registered. If this function returns OK then the caller expected
>>>> -        * to call teedev_ctx_get() or clear shm->ctx in case it's not
>>>> -        * needed any longer.
>>>> -        */
>>>>         shm->ctx = ctx;
>>>>
>>>>         rc = teedev->pool->ops->alloc(teedev->pool, shm, size, align);
>>>> @@ -112,7 +103,6 @@ static struct tee_shm *shm_alloc_helper(struct tee_context *ctx, size_t size,
>>>>         list_add_tail(&shm->link, &ctx->list_shm);
>>>>         mutex_unlock(&teedev->mutex);
>>>>
>>>> -       teedev_ctx_get(ctx);
>>>>         return shm;
>>>>  err_kfree:
>>>>         kfree(shm);
>>>> @@ -295,12 +285,10 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
>>>>                 goto err_dev_put;
>>>>         }
>>>>
>>>> -       teedev_ctx_get(ctx);
>>>> -
>>>>         shm = kzalloc(sizeof(*shm), GFP_KERNEL);
>>>>         if (!shm) {
>>>>                 ret = ERR_PTR(-ENOMEM);
>>>> -               goto err_ctx_put;
>>>> +               goto err_dev_put;
>>>>         }
>>>>
>>>>         refcount_set(&shm->refcount, 1);
>>>> @@ -313,7 +301,7 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
>>>>         num_pages = iov_iter_npages(iter, INT_MAX);
>>>>         if (!num_pages) {
>>>>                 ret = ERR_PTR(-ENOMEM);
>>>> -               goto err_ctx_put;
>>>> +               goto err_dev_put;
>>>>         }
>>>>
>>>>         shm->pages = kcalloc(num_pages, sizeof(*shm->pages), GFP_KERNEL);
>>>> @@ -361,8 +349,6 @@ register_shm_helper(struct tee_context *ctx, struct iov_iter *iter, u32 flags,
>>>>         kfree(shm->pages);
>>>>  err_free_shm:
>>>>         kfree(shm);
>>>> -err_ctx_put:
>>>> -       teedev_ctx_put(ctx);
>>>>  err_dev_put:
>>>>         tee_device_put(teedev);
>>>>         return ret;
>>>> diff --git a/include/linux/tee_core.h b/include/linux/tee_core.h
>>>> index a38494d6b5f4..13393ddac530 100644
>>>> --- a/include/linux/tee_core.h
>>>> +++ b/include/linux/tee_core.h
>>>> @@ -44,6 +44,7 @@
>>>>   * @idr:       register of user space shared memory objects allocated or
>>>>   *             registered on this device
>>>>   * @pool:      shared memory pool
>>>> + * @def_ctx:   default context used if there is no context available, e.g. internal driver calls.
>>>>   */
>>>>  struct tee_device {
>>>>         char name[TEE_MAX_DEV_NAME_LEN];
>>>> @@ -60,6 +61,7 @@ struct tee_device {
>>>>
>>>>         struct idr idr;
>>>>         struct tee_shm_pool *pool;
>>>> +       struct tee_context def_ctx;
>>>>  };
>>>>
>>>>  /**
>>>> @@ -309,6 +311,19 @@ static inline bool tee_param_is_memref(struct tee_param *param)
>>>>   */
>>>>  struct tee_context *teedev_open(struct tee_device *teedev);
>>>>
>>>> +/**
>>>> + * teedev_get_def_context() - Get default context for a struct tee_device
>>>> + * @teedev:    Device to open
>>>> + *
>>>> + * Unlike a context that returned from teedev_open(), the default context is static
>>>> + * and available as long as @teedev has a user ''other then this context''. This context
>>>> + * can be used for driver internal operation and clean up where a context should be
>>>> + * available, while tee_device_unregister() is waiting for other users to go away.
>>>> + *
>>>> + * @return a pointer to struct tee_context on success or an ERR_PTR on failure.
>>>> + */
>>>> +struct tee_context *teedev_get_def_context(struct tee_device *teedev);
>>>> +
>>>>  /**
>>>>   * teedev_close_context() - closes a struct tee_context
>>>>   * @ctx:       The struct tee_context to close
>>>> diff --git a/include/linux/tee_drv.h b/include/linux/tee_drv.h
>>>> index 1b57cddfecc8..9633e14ba484 100644
>>>> --- a/include/linux/tee_drv.h
>>>> +++ b/include/linux/tee_drv.h
>>>> @@ -7,7 +7,6 @@
>>>>  #define __TEE_DRV_H
>>>>
>>>>  #include <linux/device.h>
>>>> -#include <linux/kref.h>
>>>>  #include <linux/list.h>
>>>>  #include <linux/mod_devicetable.h>
>>>>  #include <linux/tee.h>
>>>> @@ -25,10 +24,6 @@ struct tee_device;
>>>>   * @teedev:    pointer to this drivers struct tee_device
>>>>   * @list_shm:  List of shared memory object owned by this context
>>>>   * @data:      driver specific context data, managed by the driver
>>>> - * @refcount:  reference counter for this structure
>>>> - * @releasing:  flag that indicates if context is being released right now.
>>>> - *             It is needed to break circular dependency on context during
>>>> - *              shared memory release.
>>>>   * @supp_nowait: flag that indicates that requests in this context should not
>>>>   *              wait for tee-supplicant daemon to be started if not present
>>>>   *              and just return with an error code. It is needed for requests
>>>> @@ -41,8 +36,6 @@ struct tee_context {
>>>>         struct tee_device *teedev;
>>>>         struct list_head list_shm;
>>>>         void *data;
>>>> -       struct kref refcount;
>>>> -       bool releasing;
>>>>         bool supp_nowait;
>>>>         bool cap_memref_null;
>>>>  };
>>>>
>>>> --
>>>> 2.34.1
>>>>

Re: [PATCH RFC 3/3] tee: introduce orphan tee_shm and default context

[Sumit Garg]

On Mon, 25 Nov 2024 at 03:00, Amirreza Zarrabi
<quic_azarrabi@quicinc.com> wrote:
>
>
> Hi Sumit,
>
> Thank you so much for the comemnts :).
>
> On 11/23/2024 9:32 PM, Sumit Garg wrote:
> > Hi Amirreza,
> >
> > Thanks for proposing this.
> >
> > On Fri, 22 Nov 2024 at 06:38, Amirreza Zarrabi
> > <quic_azarrabi@quicinc.com> wrote:
> >>
> >>
> >> On 11/21/2024 11:08 PM, Jens Wiklander wrote:
> >>
> >> Hi Jens,
> >>
> >>> Hi Amirreza,
> >>>
> >>> On Thu, Nov 21, 2024 at 2:37 AM Amirreza Zarrabi
> >>> <quic_azarrabi@quicinc.com> wrote:
> >>>>
> >>>> The default context has a lifespan similar to the tee_device.
> >
> > Since it's associated with tee_device context, let's call it obvious
> > via renaming it as device context instead (s/def_ctx/dev_ctx/ in this
> > patch).
> >
>
> Make sense, I'll rename it.
>
> >>>> It is used as a context for shared memory if the context to which the
> >>>> shared memory belongs is released, making the tee_shm an orphan.
> >>>> This allows the driver implementing shm_unregister to safely make
> >>>> subsequent calls, such as to a supplicant if needed.
> >>>>
> >>>> It also enables users to free the shared memory while the driver is
> >>>> blocked on unregister_tee_device safely.
> >>>>
> >>>> Preferably, this should be used for all driver internal uses, using
> >>>> teedev_get_def_context rather than calling teedev_open.
> >
> > Makes sense to me.
> >
> >>>>
> >>>> Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
> >>>> ---
> >>>>  drivers/tee/optee/core.c    |  2 +-
> >>>>  drivers/tee/optee/ffa_abi.c |  2 +-
> >>>>  drivers/tee/optee/smc_abi.c |  2 +-
> >>>>  drivers/tee/tee_core.c      | 83 +++++++++++++++++++++++++++++----------------
> >>>>  drivers/tee/tee_private.h   |  3 --
> >>>>  drivers/tee/tee_shm.c       | 18 ++--------
> >>>>  include/linux/tee_core.h    | 15 ++++++++
> >>>>  include/linux/tee_drv.h     |  7 ----
> >>>>  8 files changed, 73 insertions(+), 59 deletions(-)
> >>>>
> >>>> diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
> >>>> index c75fddc83576..78d43d0c8014 100644
> >>>> --- a/drivers/tee/optee/core.c
> >>>> +++ b/drivers/tee/optee/core.c
> >>>> @@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
> >>>>
> >>>>         optee_notif_uninit(optee);
> >>>>         optee_shm_arg_cache_uninit(optee);
> >>>> -       teedev_close_context(optee->ctx);
> >>>> +
> >>>>         /*
> >>>>          * The two devices have to be unregistered before we can free the
> >>>>          * other resources.
> >>>> diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
> >>>> index f3af5666bb11..6ad94f0788ad 100644
> >>>> --- a/drivers/tee/optee/ffa_abi.c
> >>>> +++ b/drivers/tee/optee/ffa_abi.c
> >>>> @@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
> >>>>         optee_shm_arg_cache_init(optee, arg_cache_flags);
> >>>>         mutex_init(&optee->rpmb_dev_mutex);
> >>>>         ffa_dev_set_drvdata(ffa_dev, optee);
> >>>> -       ctx = teedev_open(optee->teedev);
> >>>> +       ctx = teedev_get_def_context(optee->teedev);
> >>>>         if (IS_ERR(ctx)) {
> >>>>                 rc = PTR_ERR(ctx);
> >>>>                 goto err_rhashtable_free;
> >>>> diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
> >>>> index e9456e3e74cc..c77a3e631d04 100644
> >>>> --- a/drivers/tee/optee/smc_abi.c
> >>>> +++ b/drivers/tee/optee/smc_abi.c
> >>>> @@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
> >>>>         mutex_init(&optee->rpmb_dev_mutex);
> >>>>
> >>>>         platform_set_drvdata(pdev, optee);
> >>>> -       ctx = teedev_open(optee->teedev);
> >>>> +       ctx = teedev_get_def_context(optee->teedev);
> >>>>         if (IS_ERR(ctx)) {
> >>>>                 rc = PTR_ERR(ctx);
> >>>>                 goto err_supp_uninit;
> >>>> diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
> >>>> index 93f3b330aec8..805e1336089d 100644
> >>>> --- a/drivers/tee/tee_core.c
> >>>> +++ b/drivers/tee/tee_core.c
> >>>> @@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> >>>>                 goto err;
> >>>>         }
> >>>>
> >>>> -       kref_init(&ctx->refcount);
> >>>>         ctx->teedev = teedev;
> >>>>         INIT_LIST_HEAD(&ctx->list_shm);
> >>>>         rc = teedev->desc->ops->open(ctx);
> >>>> @@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> >>>>  }
> >>>>  EXPORT_SYMBOL_GPL(teedev_open);
> >>>>
> >>>> -void teedev_ctx_get(struct tee_context *ctx)
> >>>> +struct tee_context *teedev_get_def_context(struct tee_device *teedev)
> >>>>  {
> >>>> -       if (ctx->releasing)
> >>>> -               return;
> >>>> +       int rc;
> >>>> +       struct tee_context *ctx = &teedev->def_ctx;
> >>>>
> >>>> -       kref_get(&ctx->refcount);
> >>>> -}
> >>>> +       ctx->teedev = teedev;
> >>>> +       INIT_LIST_HEAD(&ctx->list_shm);
> >>>> +       rc = teedev->desc->ops->open(ctx);
> >>>> +       if (rc)
> >>>> +               return ERR_PTR(rc);
> >>>
> >>> I think ctx->teedev and ctx->list_shm must always be initialized or
> >>> &teedev->def_ctx can't be used in teedev_close_context().
> >>
> >> True, but &teedev->def_ctx is never used in teedev_close_context().
> >> The closing of the &teedev->def_ctx simply ignored. So once opened,
> >> &teedev->def_ctx will always remain open until the tee_device is alive.
> >>
> >>> We could initialize teedev->def_ctx on the first call to teedev_open()
> >>> on that tee_device. We need a way to tell the
> >>> teedev->desc->ops->open() to the backed driver that it's initializing
> >>> the default context though, or optee_open() can't handle the
> >>> tee-supplicant case properly.
> >>>
> >>
> >> That's a good point. This way, it is guaranteed that there is one def_ctx
> >> per teedev. There should be a way to tell the open() callback that it is
> >> a def_ctx, so it is not registered as a supplicant context.
> >>
> >>
> >>> Should we allow this function to be called more than once for each teedev?
> >>
> >> Yes, moving to teedev_open() will fix the issue.
> >>
> >>> Do we need serialization in this function if it's called after the
> >>> driver is probed?
> >>>
> >>
> >> True. I'll make sure there is no race.
> >>
> >>>>
> >>>> -static void teedev_ctx_release(struct kref *ref)
> >>>> -{
> >>>> -       struct tee_context *ctx = container_of(ref, struct tee_context,
> >>>> -                                              refcount);
> >>>> -       ctx->releasing = true;
> >>>> -       ctx->teedev->desc->ops->release(ctx);
> >>>> -       kfree(ctx);
> >>>> +       return ctx;
> >>>>  }
> >>>> +EXPORT_SYMBOL_GPL(teedev_get_def_context);
> >>>>
> >>>> -void teedev_ctx_put(struct tee_context *ctx)
> >>>> +void teedev_close_context(struct tee_context *ctx)
> >>>>  {
> >>>> -       if (ctx->releasing)
> >>>> +       struct tee_device *teedev = ctx->teedev;
> >>>> +       struct tee_shm *shm;
> >>>> +
> >>>> +       if (ctx == &teedev->def_ctx)
> >>>>                 return;
> >>>>
> >>>> -       kref_put(&ctx->refcount, teedev_ctx_release);
> >>>> -}
> >>>> +       teedev->desc->ops->release(ctx);
> >>>>
> >>>> -void teedev_close_context(struct tee_context *ctx)
> >>>> -{
> >>>> -       struct tee_device *teedev = ctx->teedev;
> >>>> +       mutex_lock(&teedev->mutex);
> >>>> +       list_for_each_entry(shm, &ctx->list_shm, link) {
> >>>> +               /* Context released. However, shm still holding a teedev reference.
> >>>> +                * Replace shm->ctx with the default context so that tee_shm_get_from_id()
> >>>> +                * fails (i.e. it is not accessible from userspace) but shm still
> >>>> +                * holds a valid context for further clean up, e.g. shm_unregister().
> >>>> +                */
> >>>
> >>> /*
> >>>  * Please format
> >>>  * multiline comments
> >>>  * like this. Please
> >>>  * keep the lines at
> >>>  * max 80 columns
> >>>  * here and at other
> >>>  * places in the patch-
> >>>  * set.
> >>>  */
> >>>
> >>
> >> Ack.
> >>
> >>>> +               shm->ctx = &teedev->def_ctx;
> >>>
> >>> shm->ctx will always point to a valid context, even if it is the
> >>> default context. It seems that we can always get hold of the correct
> >>> teedev via shm->ctx->teedev. Do we need "tee: revert removal of
> >>> redundant teedev in struct tee_shm"?
> >>>
> >>
> >> It was there in case we wanted to use NULL, but with def_ctx, it is not
> >> necessary. I am withdrawing that commit. :).
> >>
> >>> Shouldn't the shm be removed from the ctx->list_shm and be moved to
> >>> teedev->def_ctx.list_shm?
> >
> > +1
> >
>
> Ack.
>
> >>>
> >>
> >> Not really. If we put shm in the teedev->def_ctx.list_shm, by the time
> >> we are closing the def_ctx, the list is guaranteed to be empty.
> >>
> >> However, I understand it is cleaner and more consistent to do that rather
> >> than making changes to tee_shm_put().
> >>
> >> I'll do it.
> >>
> >>>> +       }
> >>>> +       mutex_unlock(&teedev->mutex);
> >>>>
> >>>> -       teedev_ctx_put(ctx);
> >>>> +       kfree(ctx);
> >>>>         tee_device_put(teedev);
> >>>>  }
> >>>>  EXPORT_SYMBOL_GPL(teedev_close_context);
> >>>> @@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
> >>>>
> >>>>         teedev->desc = teedesc;
> >>>>         teedev->pool = pool;
> >>>> +       /* Only open default context when teedev_get_def_context() called. */
> >>>> +       teedev->def_ctx.teedev = NULL;
> >
> > Why don't you open the device context here only? This will associate
> > it automatically with teedev lifespan and then
> > teedev_get_def_context() will just return a reference to that.
> >
> > -Sumit
> >
>
> So my assumption is that the tee_devic_alloc() is called as part of
> the driver initialization; there is no guarantee that at this time the
> driver is actually ready to accept any open() callback.
>

The drivers should be able to handle open() callback since we already
check for !teedesc->ops->open in the beginning of tee_devic_alloc().
Also, we need to open a device context for !TEE_DESC_PRIVILEGED such
that we don't open a supplicant device context there.

-Sumit

Re: [PATCH RFC 3/3] tee: introduce orphan tee_shm and default context

[Jens Wiklander]

On Mon, Nov 25, 2024 at 7:14 AM Sumit Garg <sumit.garg@linaro.org> wrote:
>
> On Mon, 25 Nov 2024 at 03:00, Amirreza Zarrabi
> <quic_azarrabi@quicinc.com> wrote:
> >
> >
> > Hi Sumit,
> >
> > Thank you so much for the comemnts :).
> >
> > On 11/23/2024 9:32 PM, Sumit Garg wrote:
> > > Hi Amirreza,
> > >
> > > Thanks for proposing this.
> > >
> > > On Fri, 22 Nov 2024 at 06:38, Amirreza Zarrabi
> > > <quic_azarrabi@quicinc.com> wrote:
> > >>
> > >>
> > >> On 11/21/2024 11:08 PM, Jens Wiklander wrote:
> > >>
> > >> Hi Jens,
> > >>
> > >>> Hi Amirreza,
> > >>>
> > >>> On Thu, Nov 21, 2024 at 2:37 AM Amirreza Zarrabi
> > >>> <quic_azarrabi@quicinc.com> wrote:
> > >>>>
> > >>>> The default context has a lifespan similar to the tee_device.
> > >
> > > Since it's associated with tee_device context, let's call it obvious
> > > via renaming it as device context instead (s/def_ctx/dev_ctx/ in this
> > > patch).
> > >
> >
> > Make sense, I'll rename it.
> >
> > >>>> It is used as a context for shared memory if the context to which the
> > >>>> shared memory belongs is released, making the tee_shm an orphan.
> > >>>> This allows the driver implementing shm_unregister to safely make
> > >>>> subsequent calls, such as to a supplicant if needed.
> > >>>>
> > >>>> It also enables users to free the shared memory while the driver is
> > >>>> blocked on unregister_tee_device safely.
> > >>>>
> > >>>> Preferably, this should be used for all driver internal uses, using
> > >>>> teedev_get_def_context rather than calling teedev_open.
> > >
> > > Makes sense to me.
> > >
> > >>>>
> > >>>> Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
> > >>>> ---
> > >>>>  drivers/tee/optee/core.c    |  2 +-
> > >>>>  drivers/tee/optee/ffa_abi.c |  2 +-
> > >>>>  drivers/tee/optee/smc_abi.c |  2 +-
> > >>>>  drivers/tee/tee_core.c      | 83 +++++++++++++++++++++++++++++----------------
> > >>>>  drivers/tee/tee_private.h   |  3 --
> > >>>>  drivers/tee/tee_shm.c       | 18 ++--------
> > >>>>  include/linux/tee_core.h    | 15 ++++++++
> > >>>>  include/linux/tee_drv.h     |  7 ----
> > >>>>  8 files changed, 73 insertions(+), 59 deletions(-)
> > >>>>
> > >>>> diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
> > >>>> index c75fddc83576..78d43d0c8014 100644
> > >>>> --- a/drivers/tee/optee/core.c
> > >>>> +++ b/drivers/tee/optee/core.c
> > >>>> @@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
> > >>>>
> > >>>>         optee_notif_uninit(optee);
> > >>>>         optee_shm_arg_cache_uninit(optee);
> > >>>> -       teedev_close_context(optee->ctx);
> > >>>> +
> > >>>>         /*
> > >>>>          * The two devices have to be unregistered before we can free the
> > >>>>          * other resources.
> > >>>> diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
> > >>>> index f3af5666bb11..6ad94f0788ad 100644
> > >>>> --- a/drivers/tee/optee/ffa_abi.c
> > >>>> +++ b/drivers/tee/optee/ffa_abi.c
> > >>>> @@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
> > >>>>         optee_shm_arg_cache_init(optee, arg_cache_flags);
> > >>>>         mutex_init(&optee->rpmb_dev_mutex);
> > >>>>         ffa_dev_set_drvdata(ffa_dev, optee);
> > >>>> -       ctx = teedev_open(optee->teedev);
> > >>>> +       ctx = teedev_get_def_context(optee->teedev);
> > >>>>         if (IS_ERR(ctx)) {
> > >>>>                 rc = PTR_ERR(ctx);
> > >>>>                 goto err_rhashtable_free;
> > >>>> diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
> > >>>> index e9456e3e74cc..c77a3e631d04 100644
> > >>>> --- a/drivers/tee/optee/smc_abi.c
> > >>>> +++ b/drivers/tee/optee/smc_abi.c
> > >>>> @@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
> > >>>>         mutex_init(&optee->rpmb_dev_mutex);
> > >>>>
> > >>>>         platform_set_drvdata(pdev, optee);
> > >>>> -       ctx = teedev_open(optee->teedev);
> > >>>> +       ctx = teedev_get_def_context(optee->teedev);
> > >>>>         if (IS_ERR(ctx)) {
> > >>>>                 rc = PTR_ERR(ctx);
> > >>>>                 goto err_supp_uninit;
> > >>>> diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
> > >>>> index 93f3b330aec8..805e1336089d 100644
> > >>>> --- a/drivers/tee/tee_core.c
> > >>>> +++ b/drivers/tee/tee_core.c
> > >>>> @@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> > >>>>                 goto err;
> > >>>>         }
> > >>>>
> > >>>> -       kref_init(&ctx->refcount);
> > >>>>         ctx->teedev = teedev;
> > >>>>         INIT_LIST_HEAD(&ctx->list_shm);
> > >>>>         rc = teedev->desc->ops->open(ctx);
> > >>>> @@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> > >>>>  }
> > >>>>  EXPORT_SYMBOL_GPL(teedev_open);
> > >>>>
> > >>>> -void teedev_ctx_get(struct tee_context *ctx)
> > >>>> +struct tee_context *teedev_get_def_context(struct tee_device *teedev)
> > >>>>  {
> > >>>> -       if (ctx->releasing)
> > >>>> -               return;
> > >>>> +       int rc;
> > >>>> +       struct tee_context *ctx = &teedev->def_ctx;
> > >>>>
> > >>>> -       kref_get(&ctx->refcount);
> > >>>> -}
> > >>>> +       ctx->teedev = teedev;
> > >>>> +       INIT_LIST_HEAD(&ctx->list_shm);
> > >>>> +       rc = teedev->desc->ops->open(ctx);
> > >>>> +       if (rc)
> > >>>> +               return ERR_PTR(rc);
> > >>>
> > >>> I think ctx->teedev and ctx->list_shm must always be initialized or
> > >>> &teedev->def_ctx can't be used in teedev_close_context().
> > >>
> > >> True, but &teedev->def_ctx is never used in teedev_close_context().
> > >> The closing of the &teedev->def_ctx simply ignored. So once opened,
> > >> &teedev->def_ctx will always remain open until the tee_device is alive.
> > >>
> > >>> We could initialize teedev->def_ctx on the first call to teedev_open()
> > >>> on that tee_device. We need a way to tell the
> > >>> teedev->desc->ops->open() to the backed driver that it's initializing
> > >>> the default context though, or optee_open() can't handle the
> > >>> tee-supplicant case properly.
> > >>>
> > >>
> > >> That's a good point. This way, it is guaranteed that there is one def_ctx
> > >> per teedev. There should be a way to tell the open() callback that it is
> > >> a def_ctx, so it is not registered as a supplicant context.
> > >>
> > >>
> > >>> Should we allow this function to be called more than once for each teedev?
> > >>
> > >> Yes, moving to teedev_open() will fix the issue.
> > >>
> > >>> Do we need serialization in this function if it's called after the
> > >>> driver is probed?
> > >>>
> > >>
> > >> True. I'll make sure there is no race.
> > >>
> > >>>>
> > >>>> -static void teedev_ctx_release(struct kref *ref)
> > >>>> -{
> > >>>> -       struct tee_context *ctx = container_of(ref, struct tee_context,
> > >>>> -                                              refcount);
> > >>>> -       ctx->releasing = true;
> > >>>> -       ctx->teedev->desc->ops->release(ctx);
> > >>>> -       kfree(ctx);
> > >>>> +       return ctx;
> > >>>>  }
> > >>>> +EXPORT_SYMBOL_GPL(teedev_get_def_context);
> > >>>>
> > >>>> -void teedev_ctx_put(struct tee_context *ctx)
> > >>>> +void teedev_close_context(struct tee_context *ctx)
> > >>>>  {
> > >>>> -       if (ctx->releasing)
> > >>>> +       struct tee_device *teedev = ctx->teedev;
> > >>>> +       struct tee_shm *shm;
> > >>>> +
> > >>>> +       if (ctx == &teedev->def_ctx)
> > >>>>                 return;
> > >>>>
> > >>>> -       kref_put(&ctx->refcount, teedev_ctx_release);
> > >>>> -}
> > >>>> +       teedev->desc->ops->release(ctx);
> > >>>>
> > >>>> -void teedev_close_context(struct tee_context *ctx)
> > >>>> -{
> > >>>> -       struct tee_device *teedev = ctx->teedev;
> > >>>> +       mutex_lock(&teedev->mutex);
> > >>>> +       list_for_each_entry(shm, &ctx->list_shm, link) {
> > >>>> +               /* Context released. However, shm still holding a teedev reference.
> > >>>> +                * Replace shm->ctx with the default context so that tee_shm_get_from_id()
> > >>>> +                * fails (i.e. it is not accessible from userspace) but shm still
> > >>>> +                * holds a valid context for further clean up, e.g. shm_unregister().
> > >>>> +                */
> > >>>
> > >>> /*
> > >>>  * Please format
> > >>>  * multiline comments
> > >>>  * like this. Please
> > >>>  * keep the lines at
> > >>>  * max 80 columns
> > >>>  * here and at other
> > >>>  * places in the patch-
> > >>>  * set.
> > >>>  */
> > >>>
> > >>
> > >> Ack.
> > >>
> > >>>> +               shm->ctx = &teedev->def_ctx;
> > >>>
> > >>> shm->ctx will always point to a valid context, even if it is the
> > >>> default context. It seems that we can always get hold of the correct
> > >>> teedev via shm->ctx->teedev. Do we need "tee: revert removal of
> > >>> redundant teedev in struct tee_shm"?
> > >>>
> > >>
> > >> It was there in case we wanted to use NULL, but with def_ctx, it is not
> > >> necessary. I am withdrawing that commit. :).
> > >>
> > >>> Shouldn't the shm be removed from the ctx->list_shm and be moved to
> > >>> teedev->def_ctx.list_shm?
> > >
> > > +1
> > >
> >
> > Ack.
> >
> > >>>
> > >>
> > >> Not really. If we put shm in the teedev->def_ctx.list_shm, by the time
> > >> we are closing the def_ctx, the list is guaranteed to be empty.
> > >>
> > >> However, I understand it is cleaner and more consistent to do that rather
> > >> than making changes to tee_shm_put().
> > >>
> > >> I'll do it.
> > >>
> > >>>> +       }
> > >>>> +       mutex_unlock(&teedev->mutex);
> > >>>>
> > >>>> -       teedev_ctx_put(ctx);
> > >>>> +       kfree(ctx);
> > >>>>         tee_device_put(teedev);
> > >>>>  }
> > >>>>  EXPORT_SYMBOL_GPL(teedev_close_context);
> > >>>> @@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
> > >>>>
> > >>>>         teedev->desc = teedesc;
> > >>>>         teedev->pool = pool;
> > >>>> +       /* Only open default context when teedev_get_def_context() called. */
> > >>>> +       teedev->def_ctx.teedev = NULL;
> > >
> > > Why don't you open the device context here only? This will associate
> > > it automatically with teedev lifespan and then
> > > teedev_get_def_context() will just return a reference to that.
> > >
> > > -Sumit
> > >
> >
> > So my assumption is that the tee_devic_alloc() is called as part of
> > the driver initialization; there is no guarantee that at this time the
> > driver is actually ready to accept any open() callback.
> >
>
> The drivers should be able to handle open() callback since we already
> check for !teedesc->ops->open in the beginning of tee_devic_alloc().
> Also, we need to open a device context for !TEE_DESC_PRIVILEGED such
> that we don't open a supplicant device context there.

It would be nice to have the device context fully initialized when the
probe function returns. How about adding a "bool is_dev_ctx" to struct
tee_context so the open() callback can tell that this is a special
tee_contex?

Cheers,
Jens

>
> -Sumit

Re: [PATCH RFC 3/3] tee: introduce orphan tee_shm and default context

[Sumit Garg]

On Mon, 25 Nov 2024 at 12:53, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>
> On Mon, Nov 25, 2024 at 7:14 AM Sumit Garg <sumit.garg@linaro.org> wrote:
> >
> > On Mon, 25 Nov 2024 at 03:00, Amirreza Zarrabi
> > <quic_azarrabi@quicinc.com> wrote:
> > >
> > >
> > > Hi Sumit,
> > >
> > > Thank you so much for the comemnts :).
> > >
> > > On 11/23/2024 9:32 PM, Sumit Garg wrote:
> > > > Hi Amirreza,
> > > >
> > > > Thanks for proposing this.
> > > >
> > > > On Fri, 22 Nov 2024 at 06:38, Amirreza Zarrabi
> > > > <quic_azarrabi@quicinc.com> wrote:
> > > >>
> > > >>
> > > >> On 11/21/2024 11:08 PM, Jens Wiklander wrote:
> > > >>
> > > >> Hi Jens,
> > > >>
> > > >>> Hi Amirreza,
> > > >>>
> > > >>> On Thu, Nov 21, 2024 at 2:37 AM Amirreza Zarrabi
> > > >>> <quic_azarrabi@quicinc.com> wrote:
> > > >>>>
> > > >>>> The default context has a lifespan similar to the tee_device.
> > > >
> > > > Since it's associated with tee_device context, let's call it obvious
> > > > via renaming it as device context instead (s/def_ctx/dev_ctx/ in this
> > > > patch).
> > > >
> > >
> > > Make sense, I'll rename it.
> > >
> > > >>>> It is used as a context for shared memory if the context to which the
> > > >>>> shared memory belongs is released, making the tee_shm an orphan.
> > > >>>> This allows the driver implementing shm_unregister to safely make
> > > >>>> subsequent calls, such as to a supplicant if needed.
> > > >>>>
> > > >>>> It also enables users to free the shared memory while the driver is
> > > >>>> blocked on unregister_tee_device safely.
> > > >>>>
> > > >>>> Preferably, this should be used for all driver internal uses, using
> > > >>>> teedev_get_def_context rather than calling teedev_open.
> > > >
> > > > Makes sense to me.
> > > >
> > > >>>>
> > > >>>> Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
> > > >>>> ---
> > > >>>>  drivers/tee/optee/core.c    |  2 +-
> > > >>>>  drivers/tee/optee/ffa_abi.c |  2 +-
> > > >>>>  drivers/tee/optee/smc_abi.c |  2 +-
> > > >>>>  drivers/tee/tee_core.c      | 83 +++++++++++++++++++++++++++++----------------
> > > >>>>  drivers/tee/tee_private.h   |  3 --
> > > >>>>  drivers/tee/tee_shm.c       | 18 ++--------
> > > >>>>  include/linux/tee_core.h    | 15 ++++++++
> > > >>>>  include/linux/tee_drv.h     |  7 ----
> > > >>>>  8 files changed, 73 insertions(+), 59 deletions(-)
> > > >>>>
> > > >>>> diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
> > > >>>> index c75fddc83576..78d43d0c8014 100644
> > > >>>> --- a/drivers/tee/optee/core.c
> > > >>>> +++ b/drivers/tee/optee/core.c
> > > >>>> @@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
> > > >>>>
> > > >>>>         optee_notif_uninit(optee);
> > > >>>>         optee_shm_arg_cache_uninit(optee);
> > > >>>> -       teedev_close_context(optee->ctx);
> > > >>>> +
> > > >>>>         /*
> > > >>>>          * The two devices have to be unregistered before we can free the
> > > >>>>          * other resources.
> > > >>>> diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
> > > >>>> index f3af5666bb11..6ad94f0788ad 100644
> > > >>>> --- a/drivers/tee/optee/ffa_abi.c
> > > >>>> +++ b/drivers/tee/optee/ffa_abi.c
> > > >>>> @@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
> > > >>>>         optee_shm_arg_cache_init(optee, arg_cache_flags);
> > > >>>>         mutex_init(&optee->rpmb_dev_mutex);
> > > >>>>         ffa_dev_set_drvdata(ffa_dev, optee);
> > > >>>> -       ctx = teedev_open(optee->teedev);
> > > >>>> +       ctx = teedev_get_def_context(optee->teedev);
> > > >>>>         if (IS_ERR(ctx)) {
> > > >>>>                 rc = PTR_ERR(ctx);
> > > >>>>                 goto err_rhashtable_free;
> > > >>>> diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
> > > >>>> index e9456e3e74cc..c77a3e631d04 100644
> > > >>>> --- a/drivers/tee/optee/smc_abi.c
> > > >>>> +++ b/drivers/tee/optee/smc_abi.c
> > > >>>> @@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
> > > >>>>         mutex_init(&optee->rpmb_dev_mutex);
> > > >>>>
> > > >>>>         platform_set_drvdata(pdev, optee);
> > > >>>> -       ctx = teedev_open(optee->teedev);
> > > >>>> +       ctx = teedev_get_def_context(optee->teedev);
> > > >>>>         if (IS_ERR(ctx)) {
> > > >>>>                 rc = PTR_ERR(ctx);
> > > >>>>                 goto err_supp_uninit;
> > > >>>> diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
> > > >>>> index 93f3b330aec8..805e1336089d 100644
> > > >>>> --- a/drivers/tee/tee_core.c
> > > >>>> +++ b/drivers/tee/tee_core.c
> > > >>>> @@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> > > >>>>                 goto err;
> > > >>>>         }
> > > >>>>
> > > >>>> -       kref_init(&ctx->refcount);
> > > >>>>         ctx->teedev = teedev;
> > > >>>>         INIT_LIST_HEAD(&ctx->list_shm);
> > > >>>>         rc = teedev->desc->ops->open(ctx);
> > > >>>> @@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> > > >>>>  }
> > > >>>>  EXPORT_SYMBOL_GPL(teedev_open);
> > > >>>>
> > > >>>> -void teedev_ctx_get(struct tee_context *ctx)
> > > >>>> +struct tee_context *teedev_get_def_context(struct tee_device *teedev)
> > > >>>>  {
> > > >>>> -       if (ctx->releasing)
> > > >>>> -               return;
> > > >>>> +       int rc;
> > > >>>> +       struct tee_context *ctx = &teedev->def_ctx;
> > > >>>>
> > > >>>> -       kref_get(&ctx->refcount);
> > > >>>> -}
> > > >>>> +       ctx->teedev = teedev;
> > > >>>> +       INIT_LIST_HEAD(&ctx->list_shm);
> > > >>>> +       rc = teedev->desc->ops->open(ctx);
> > > >>>> +       if (rc)
> > > >>>> +               return ERR_PTR(rc);
> > > >>>
> > > >>> I think ctx->teedev and ctx->list_shm must always be initialized or
> > > >>> &teedev->def_ctx can't be used in teedev_close_context().
> > > >>
> > > >> True, but &teedev->def_ctx is never used in teedev_close_context().
> > > >> The closing of the &teedev->def_ctx simply ignored. So once opened,
> > > >> &teedev->def_ctx will always remain open until the tee_device is alive.
> > > >>
> > > >>> We could initialize teedev->def_ctx on the first call to teedev_open()
> > > >>> on that tee_device. We need a way to tell the
> > > >>> teedev->desc->ops->open() to the backed driver that it's initializing
> > > >>> the default context though, or optee_open() can't handle the
> > > >>> tee-supplicant case properly.
> > > >>>
> > > >>
> > > >> That's a good point. This way, it is guaranteed that there is one def_ctx
> > > >> per teedev. There should be a way to tell the open() callback that it is
> > > >> a def_ctx, so it is not registered as a supplicant context.
> > > >>
> > > >>
> > > >>> Should we allow this function to be called more than once for each teedev?
> > > >>
> > > >> Yes, moving to teedev_open() will fix the issue.
> > > >>
> > > >>> Do we need serialization in this function if it's called after the
> > > >>> driver is probed?
> > > >>>
> > > >>
> > > >> True. I'll make sure there is no race.
> > > >>
> > > >>>>
> > > >>>> -static void teedev_ctx_release(struct kref *ref)
> > > >>>> -{
> > > >>>> -       struct tee_context *ctx = container_of(ref, struct tee_context,
> > > >>>> -                                              refcount);
> > > >>>> -       ctx->releasing = true;
> > > >>>> -       ctx->teedev->desc->ops->release(ctx);
> > > >>>> -       kfree(ctx);
> > > >>>> +       return ctx;
> > > >>>>  }
> > > >>>> +EXPORT_SYMBOL_GPL(teedev_get_def_context);
> > > >>>>
> > > >>>> -void teedev_ctx_put(struct tee_context *ctx)
> > > >>>> +void teedev_close_context(struct tee_context *ctx)
> > > >>>>  {
> > > >>>> -       if (ctx->releasing)
> > > >>>> +       struct tee_device *teedev = ctx->teedev;
> > > >>>> +       struct tee_shm *shm;
> > > >>>> +
> > > >>>> +       if (ctx == &teedev->def_ctx)
> > > >>>>                 return;
> > > >>>>
> > > >>>> -       kref_put(&ctx->refcount, teedev_ctx_release);
> > > >>>> -}
> > > >>>> +       teedev->desc->ops->release(ctx);
> > > >>>>
> > > >>>> -void teedev_close_context(struct tee_context *ctx)
> > > >>>> -{
> > > >>>> -       struct tee_device *teedev = ctx->teedev;
> > > >>>> +       mutex_lock(&teedev->mutex);
> > > >>>> +       list_for_each_entry(shm, &ctx->list_shm, link) {
> > > >>>> +               /* Context released. However, shm still holding a teedev reference.
> > > >>>> +                * Replace shm->ctx with the default context so that tee_shm_get_from_id()
> > > >>>> +                * fails (i.e. it is not accessible from userspace) but shm still
> > > >>>> +                * holds a valid context for further clean up, e.g. shm_unregister().
> > > >>>> +                */
> > > >>>
> > > >>> /*
> > > >>>  * Please format
> > > >>>  * multiline comments
> > > >>>  * like this. Please
> > > >>>  * keep the lines at
> > > >>>  * max 80 columns
> > > >>>  * here and at other
> > > >>>  * places in the patch-
> > > >>>  * set.
> > > >>>  */
> > > >>>
> > > >>
> > > >> Ack.
> > > >>
> > > >>>> +               shm->ctx = &teedev->def_ctx;
> > > >>>
> > > >>> shm->ctx will always point to a valid context, even if it is the
> > > >>> default context. It seems that we can always get hold of the correct
> > > >>> teedev via shm->ctx->teedev. Do we need "tee: revert removal of
> > > >>> redundant teedev in struct tee_shm"?
> > > >>>
> > > >>
> > > >> It was there in case we wanted to use NULL, but with def_ctx, it is not
> > > >> necessary. I am withdrawing that commit. :).
> > > >>
> > > >>> Shouldn't the shm be removed from the ctx->list_shm and be moved to
> > > >>> teedev->def_ctx.list_shm?
> > > >
> > > > +1
> > > >
> > >
> > > Ack.
> > >
> > > >>>
> > > >>
> > > >> Not really. If we put shm in the teedev->def_ctx.list_shm, by the time
> > > >> we are closing the def_ctx, the list is guaranteed to be empty.
> > > >>
> > > >> However, I understand it is cleaner and more consistent to do that rather
> > > >> than making changes to tee_shm_put().
> > > >>
> > > >> I'll do it.
> > > >>
> > > >>>> +       }
> > > >>>> +       mutex_unlock(&teedev->mutex);
> > > >>>>
> > > >>>> -       teedev_ctx_put(ctx);
> > > >>>> +       kfree(ctx);
> > > >>>>         tee_device_put(teedev);
> > > >>>>  }
> > > >>>>  EXPORT_SYMBOL_GPL(teedev_close_context);
> > > >>>> @@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
> > > >>>>
> > > >>>>         teedev->desc = teedesc;
> > > >>>>         teedev->pool = pool;
> > > >>>> +       /* Only open default context when teedev_get_def_context() called. */
> > > >>>> +       teedev->def_ctx.teedev = NULL;
> > > >
> > > > Why don't you open the device context here only? This will associate
> > > > it automatically with teedev lifespan and then
> > > > teedev_get_def_context() will just return a reference to that.
> > > >
> > > > -Sumit
> > > >
> > >
> > > So my assumption is that the tee_devic_alloc() is called as part of
> > > the driver initialization; there is no guarantee that at this time the
> > > driver is actually ready to accept any open() callback.
> > >
> >
> > The drivers should be able to handle open() callback since we already
> > check for !teedesc->ops->open in the beginning of tee_devic_alloc().
> > Also, we need to open a device context for !TEE_DESC_PRIVILEGED such
> > that we don't open a supplicant device context there.
>
> It would be nice to have the device context fully initialized when the
> probe function returns. How about adding a "bool is_dev_ctx" to struct
> tee_context so the open() callback can tell that this is a special
> tee_contex?

Sure, that will be useful to distinguish the device context from
normal client context.

-Sumit

>
> Cheers,
> Jens
>
> >
> > -Sumit

Re: [PATCH RFC 3/3] tee: introduce orphan tee_shm and default context

[Amirreza Zarrabi]

On 11/25/2024 6:51 PM, Sumit Garg wrote:
> On Mon, 25 Nov 2024 at 12:53, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>
>> On Mon, Nov 25, 2024 at 7:14 AM Sumit Garg <sumit.garg@linaro.org> wrote:
>>>
>>> On Mon, 25 Nov 2024 at 03:00, Amirreza Zarrabi
>>> <quic_azarrabi@quicinc.com> wrote:
>>>>
>>>>
>>>> Hi Sumit,
>>>>
>>>> Thank you so much for the comemnts :).
>>>>
>>>> On 11/23/2024 9:32 PM, Sumit Garg wrote:
>>>>> Hi Amirreza,
>>>>>
>>>>> Thanks for proposing this.
>>>>>
>>>>> On Fri, 22 Nov 2024 at 06:38, Amirreza Zarrabi
>>>>> <quic_azarrabi@quicinc.com> wrote:
>>>>>>
>>>>>>
>>>>>> On 11/21/2024 11:08 PM, Jens Wiklander wrote:
>>>>>>
>>>>>> Hi Jens,
>>>>>>
>>>>>>> Hi Amirreza,
>>>>>>>
>>>>>>> On Thu, Nov 21, 2024 at 2:37 AM Amirreza Zarrabi
>>>>>>> <quic_azarrabi@quicinc.com> wrote:
>>>>>>>>
>>>>>>>> The default context has a lifespan similar to the tee_device.
>>>>>
>>>>> Since it's associated with tee_device context, let's call it obvious
>>>>> via renaming it as device context instead (s/def_ctx/dev_ctx/ in this
>>>>> patch).
>>>>>
>>>>
>>>> Make sense, I'll rename it.
>>>>
>>>>>>>> It is used as a context for shared memory if the context to which the
>>>>>>>> shared memory belongs is released, making the tee_shm an orphan.
>>>>>>>> This allows the driver implementing shm_unregister to safely make
>>>>>>>> subsequent calls, such as to a supplicant if needed.
>>>>>>>>
>>>>>>>> It also enables users to free the shared memory while the driver is
>>>>>>>> blocked on unregister_tee_device safely.
>>>>>>>>
>>>>>>>> Preferably, this should be used for all driver internal uses, using
>>>>>>>> teedev_get_def_context rather than calling teedev_open.
>>>>>
>>>>> Makes sense to me.
>>>>>
>>>>>>>>
>>>>>>>> Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
>>>>>>>> ---
>>>>>>>>  drivers/tee/optee/core.c    |  2 +-
>>>>>>>>  drivers/tee/optee/ffa_abi.c |  2 +-
>>>>>>>>  drivers/tee/optee/smc_abi.c |  2 +-
>>>>>>>>  drivers/tee/tee_core.c      | 83 +++++++++++++++++++++++++++++----------------
>>>>>>>>  drivers/tee/tee_private.h   |  3 --
>>>>>>>>  drivers/tee/tee_shm.c       | 18 ++--------
>>>>>>>>  include/linux/tee_core.h    | 15 ++++++++
>>>>>>>>  include/linux/tee_drv.h     |  7 ----
>>>>>>>>  8 files changed, 73 insertions(+), 59 deletions(-)
>>>>>>>>
>>>>>>>> diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
>>>>>>>> index c75fddc83576..78d43d0c8014 100644
>>>>>>>> --- a/drivers/tee/optee/core.c
>>>>>>>> +++ b/drivers/tee/optee/core.c
>>>>>>>> @@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
>>>>>>>>
>>>>>>>>         optee_notif_uninit(optee);
>>>>>>>>         optee_shm_arg_cache_uninit(optee);
>>>>>>>> -       teedev_close_context(optee->ctx);
>>>>>>>> +
>>>>>>>>         /*
>>>>>>>>          * The two devices have to be unregistered before we can free the
>>>>>>>>          * other resources.
>>>>>>>> diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
>>>>>>>> index f3af5666bb11..6ad94f0788ad 100644
>>>>>>>> --- a/drivers/tee/optee/ffa_abi.c
>>>>>>>> +++ b/drivers/tee/optee/ffa_abi.c
>>>>>>>> @@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
>>>>>>>>         optee_shm_arg_cache_init(optee, arg_cache_flags);
>>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
>>>>>>>>         ffa_dev_set_drvdata(ffa_dev, optee);
>>>>>>>> -       ctx = teedev_open(optee->teedev);
>>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
>>>>>>>>         if (IS_ERR(ctx)) {
>>>>>>>>                 rc = PTR_ERR(ctx);
>>>>>>>>                 goto err_rhashtable_free;
>>>>>>>> diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
>>>>>>>> index e9456e3e74cc..c77a3e631d04 100644
>>>>>>>> --- a/drivers/tee/optee/smc_abi.c
>>>>>>>> +++ b/drivers/tee/optee/smc_abi.c
>>>>>>>> @@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
>>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
>>>>>>>>
>>>>>>>>         platform_set_drvdata(pdev, optee);
>>>>>>>> -       ctx = teedev_open(optee->teedev);
>>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
>>>>>>>>         if (IS_ERR(ctx)) {
>>>>>>>>                 rc = PTR_ERR(ctx);
>>>>>>>>                 goto err_supp_uninit;
>>>>>>>> diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
>>>>>>>> index 93f3b330aec8..805e1336089d 100644
>>>>>>>> --- a/drivers/tee/tee_core.c
>>>>>>>> +++ b/drivers/tee/tee_core.c
>>>>>>>> @@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
>>>>>>>>                 goto err;
>>>>>>>>         }
>>>>>>>>
>>>>>>>> -       kref_init(&ctx->refcount);
>>>>>>>>         ctx->teedev = teedev;
>>>>>>>>         INIT_LIST_HEAD(&ctx->list_shm);
>>>>>>>>         rc = teedev->desc->ops->open(ctx);
>>>>>>>> @@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
>>>>>>>>  }
>>>>>>>>  EXPORT_SYMBOL_GPL(teedev_open);
>>>>>>>>
>>>>>>>> -void teedev_ctx_get(struct tee_context *ctx)
>>>>>>>> +struct tee_context *teedev_get_def_context(struct tee_device *teedev)
>>>>>>>>  {
>>>>>>>> -       if (ctx->releasing)
>>>>>>>> -               return;
>>>>>>>> +       int rc;
>>>>>>>> +       struct tee_context *ctx = &teedev->def_ctx;
>>>>>>>>
>>>>>>>> -       kref_get(&ctx->refcount);
>>>>>>>> -}
>>>>>>>> +       ctx->teedev = teedev;
>>>>>>>> +       INIT_LIST_HEAD(&ctx->list_shm);
>>>>>>>> +       rc = teedev->desc->ops->open(ctx);
>>>>>>>> +       if (rc)
>>>>>>>> +               return ERR_PTR(rc);
>>>>>>>
>>>>>>> I think ctx->teedev and ctx->list_shm must always be initialized or
>>>>>>> &teedev->def_ctx can't be used in teedev_close_context().
>>>>>>
>>>>>> True, but &teedev->def_ctx is never used in teedev_close_context().
>>>>>> The closing of the &teedev->def_ctx simply ignored. So once opened,
>>>>>> &teedev->def_ctx will always remain open until the tee_device is alive.
>>>>>>
>>>>>>> We could initialize teedev->def_ctx on the first call to teedev_open()
>>>>>>> on that tee_device. We need a way to tell the
>>>>>>> teedev->desc->ops->open() to the backed driver that it's initializing
>>>>>>> the default context though, or optee_open() can't handle the
>>>>>>> tee-supplicant case properly.
>>>>>>>
>>>>>>
>>>>>> That's a good point. This way, it is guaranteed that there is one def_ctx
>>>>>> per teedev. There should be a way to tell the open() callback that it is
>>>>>> a def_ctx, so it is not registered as a supplicant context.
>>>>>>
>>>>>>
>>>>>>> Should we allow this function to be called more than once for each teedev?
>>>>>>
>>>>>> Yes, moving to teedev_open() will fix the issue.
>>>>>>
>>>>>>> Do we need serialization in this function if it's called after the
>>>>>>> driver is probed?
>>>>>>>
>>>>>>
>>>>>> True. I'll make sure there is no race.
>>>>>>
>>>>>>>>
>>>>>>>> -static void teedev_ctx_release(struct kref *ref)
>>>>>>>> -{
>>>>>>>> -       struct tee_context *ctx = container_of(ref, struct tee_context,
>>>>>>>> -                                              refcount);
>>>>>>>> -       ctx->releasing = true;
>>>>>>>> -       ctx->teedev->desc->ops->release(ctx);
>>>>>>>> -       kfree(ctx);
>>>>>>>> +       return ctx;
>>>>>>>>  }
>>>>>>>> +EXPORT_SYMBOL_GPL(teedev_get_def_context);
>>>>>>>>
>>>>>>>> -void teedev_ctx_put(struct tee_context *ctx)
>>>>>>>> +void teedev_close_context(struct tee_context *ctx)
>>>>>>>>  {
>>>>>>>> -       if (ctx->releasing)
>>>>>>>> +       struct tee_device *teedev = ctx->teedev;
>>>>>>>> +       struct tee_shm *shm;
>>>>>>>> +
>>>>>>>> +       if (ctx == &teedev->def_ctx)
>>>>>>>>                 return;
>>>>>>>>
>>>>>>>> -       kref_put(&ctx->refcount, teedev_ctx_release);
>>>>>>>> -}
>>>>>>>> +       teedev->desc->ops->release(ctx);
>>>>>>>>
>>>>>>>> -void teedev_close_context(struct tee_context *ctx)
>>>>>>>> -{
>>>>>>>> -       struct tee_device *teedev = ctx->teedev;
>>>>>>>> +       mutex_lock(&teedev->mutex);
>>>>>>>> +       list_for_each_entry(shm, &ctx->list_shm, link) {
>>>>>>>> +               /* Context released. However, shm still holding a teedev reference.
>>>>>>>> +                * Replace shm->ctx with the default context so that tee_shm_get_from_id()
>>>>>>>> +                * fails (i.e. it is not accessible from userspace) but shm still
>>>>>>>> +                * holds a valid context for further clean up, e.g. shm_unregister().
>>>>>>>> +                */
>>>>>>>
>>>>>>> /*
>>>>>>>  * Please format
>>>>>>>  * multiline comments
>>>>>>>  * like this. Please
>>>>>>>  * keep the lines at
>>>>>>>  * max 80 columns
>>>>>>>  * here and at other
>>>>>>>  * places in the patch-
>>>>>>>  * set.
>>>>>>>  */
>>>>>>>
>>>>>>
>>>>>> Ack.
>>>>>>
>>>>>>>> +               shm->ctx = &teedev->def_ctx;
>>>>>>>
>>>>>>> shm->ctx will always point to a valid context, even if it is the
>>>>>>> default context. It seems that we can always get hold of the correct
>>>>>>> teedev via shm->ctx->teedev. Do we need "tee: revert removal of
>>>>>>> redundant teedev in struct tee_shm"?
>>>>>>>
>>>>>>
>>>>>> It was there in case we wanted to use NULL, but with def_ctx, it is not
>>>>>> necessary. I am withdrawing that commit. :).
>>>>>>
>>>>>>> Shouldn't the shm be removed from the ctx->list_shm and be moved to
>>>>>>> teedev->def_ctx.list_shm?
>>>>>
>>>>> +1
>>>>>
>>>>
>>>> Ack.
>>>>
>>>>>>>
>>>>>>
>>>>>> Not really. If we put shm in the teedev->def_ctx.list_shm, by the time
>>>>>> we are closing the def_ctx, the list is guaranteed to be empty.
>>>>>>
>>>>>> However, I understand it is cleaner and more consistent to do that rather
>>>>>> than making changes to tee_shm_put().
>>>>>>
>>>>>> I'll do it.
>>>>>>
>>>>>>>> +       }
>>>>>>>> +       mutex_unlock(&teedev->mutex);
>>>>>>>>
>>>>>>>> -       teedev_ctx_put(ctx);
>>>>>>>> +       kfree(ctx);
>>>>>>>>         tee_device_put(teedev);
>>>>>>>>  }
>>>>>>>>  EXPORT_SYMBOL_GPL(teedev_close_context);
>>>>>>>> @@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
>>>>>>>>
>>>>>>>>         teedev->desc = teedesc;
>>>>>>>>         teedev->pool = pool;
>>>>>>>> +       /* Only open default context when teedev_get_def_context() called. */
>>>>>>>> +       teedev->def_ctx.teedev = NULL;
>>>>>
>>>>> Why don't you open the device context here only? This will associate
>>>>> it automatically with teedev lifespan and then
>>>>> teedev_get_def_context() will just return a reference to that.
>>>>>
>>>>> -Sumit
>>>>>
>>>>
>>>> So my assumption is that the tee_devic_alloc() is called as part of
>>>> the driver initialization; there is no guarantee that at this time the
>>>> driver is actually ready to accept any open() callback.
>>>>
>>>
>>> The drivers should be able to handle open() callback since we already
>>> check for !teedesc->ops->open in the beginning of tee_devic_alloc().
>>> Also, we need to open a device context for !TEE_DESC_PRIVILEGED such
>>> that we don't open a supplicant device context there.
>>
>> It would be nice to have the device context fully initialized when the
>> probe function returns. How about adding a "bool is_dev_ctx" to struct
>> tee_context so the open() callback can tell that this is a special
>> tee_contex?
> 
> Sure, that will be useful to distinguish the device context from
> normal client context.
> 
> -Sumit
> 

So, as far as the open() callback, I do not believe checking if it is not null
is reasonable for calling it here. Most drivers allocate resources and then
initialize them. So, assume these steps for a TEE driver:
 (1) allocate internal data structures, 
 (2) allocate the device,
 (3) initialize the internal data structurse and then
 (4) register the device.

Having these steps for a backend driver means that if you call open() at
step (2), the internal data structures are not ready.

I was originally thinking of going with Jens' suggestion to open dev_ctx in
the teedev_open(), and use a flag to distinguish the type of context for
the open() callback

What about this:
Open the dev_ctx in the tee_device_register(), at the last step before
setting the TEE_DEVICE_FLAG_REGISTERED flag. Then the open() callback can
check for this flag to determine if it is a normal context or dev_ctx.
If the open() is called while the device has not been registered, it should
handle it differently

- Amir

>>
>> Cheers,
>> Jens
>>
>>>
>>> -Sumit

Re: [PATCH RFC 3/3] tee: introduce orphan tee_shm and default context

[Jens Wiklander]

On Mon, Nov 25, 2024 at 9:55 PM Amirreza Zarrabi
<quic_azarrabi@quicinc.com> wrote:
>
>
>
> On 11/25/2024 6:51 PM, Sumit Garg wrote:
> > On Mon, 25 Nov 2024 at 12:53, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >>
> >> On Mon, Nov 25, 2024 at 7:14 AM Sumit Garg <sumit.garg@linaro.org> wrote:
> >>>
> >>> On Mon, 25 Nov 2024 at 03:00, Amirreza Zarrabi
> >>> <quic_azarrabi@quicinc.com> wrote:
> >>>>
> >>>>
> >>>> Hi Sumit,
> >>>>
> >>>> Thank you so much for the comemnts :).
> >>>>
> >>>> On 11/23/2024 9:32 PM, Sumit Garg wrote:
> >>>>> Hi Amirreza,
> >>>>>
> >>>>> Thanks for proposing this.
> >>>>>
> >>>>> On Fri, 22 Nov 2024 at 06:38, Amirreza Zarrabi
> >>>>> <quic_azarrabi@quicinc.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>> On 11/21/2024 11:08 PM, Jens Wiklander wrote:
> >>>>>>
> >>>>>> Hi Jens,
> >>>>>>
> >>>>>>> Hi Amirreza,
> >>>>>>>
> >>>>>>> On Thu, Nov 21, 2024 at 2:37 AM Amirreza Zarrabi
> >>>>>>> <quic_azarrabi@quicinc.com> wrote:
> >>>>>>>>
> >>>>>>>> The default context has a lifespan similar to the tee_device.
> >>>>>
> >>>>> Since it's associated with tee_device context, let's call it obvious
> >>>>> via renaming it as device context instead (s/def_ctx/dev_ctx/ in this
> >>>>> patch).
> >>>>>
> >>>>
> >>>> Make sense, I'll rename it.
> >>>>
> >>>>>>>> It is used as a context for shared memory if the context to which the
> >>>>>>>> shared memory belongs is released, making the tee_shm an orphan.
> >>>>>>>> This allows the driver implementing shm_unregister to safely make
> >>>>>>>> subsequent calls, such as to a supplicant if needed.
> >>>>>>>>
> >>>>>>>> It also enables users to free the shared memory while the driver is
> >>>>>>>> blocked on unregister_tee_device safely.
> >>>>>>>>
> >>>>>>>> Preferably, this should be used for all driver internal uses, using
> >>>>>>>> teedev_get_def_context rather than calling teedev_open.
> >>>>>
> >>>>> Makes sense to me.
> >>>>>
> >>>>>>>>
> >>>>>>>> Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
> >>>>>>>> ---
> >>>>>>>>  drivers/tee/optee/core.c    |  2 +-
> >>>>>>>>  drivers/tee/optee/ffa_abi.c |  2 +-
> >>>>>>>>  drivers/tee/optee/smc_abi.c |  2 +-
> >>>>>>>>  drivers/tee/tee_core.c      | 83 +++++++++++++++++++++++++++++----------------
> >>>>>>>>  drivers/tee/tee_private.h   |  3 --
> >>>>>>>>  drivers/tee/tee_shm.c       | 18 ++--------
> >>>>>>>>  include/linux/tee_core.h    | 15 ++++++++
> >>>>>>>>  include/linux/tee_drv.h     |  7 ----
> >>>>>>>>  8 files changed, 73 insertions(+), 59 deletions(-)
> >>>>>>>>
> >>>>>>>> diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
> >>>>>>>> index c75fddc83576..78d43d0c8014 100644
> >>>>>>>> --- a/drivers/tee/optee/core.c
> >>>>>>>> +++ b/drivers/tee/optee/core.c
> >>>>>>>> @@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
> >>>>>>>>
> >>>>>>>>         optee_notif_uninit(optee);
> >>>>>>>>         optee_shm_arg_cache_uninit(optee);
> >>>>>>>> -       teedev_close_context(optee->ctx);
> >>>>>>>> +
> >>>>>>>>         /*
> >>>>>>>>          * The two devices have to be unregistered before we can free the
> >>>>>>>>          * other resources.
> >>>>>>>> diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
> >>>>>>>> index f3af5666bb11..6ad94f0788ad 100644
> >>>>>>>> --- a/drivers/tee/optee/ffa_abi.c
> >>>>>>>> +++ b/drivers/tee/optee/ffa_abi.c
> >>>>>>>> @@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
> >>>>>>>>         optee_shm_arg_cache_init(optee, arg_cache_flags);
> >>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
> >>>>>>>>         ffa_dev_set_drvdata(ffa_dev, optee);
> >>>>>>>> -       ctx = teedev_open(optee->teedev);
> >>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
> >>>>>>>>         if (IS_ERR(ctx)) {
> >>>>>>>>                 rc = PTR_ERR(ctx);
> >>>>>>>>                 goto err_rhashtable_free;
> >>>>>>>> diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
> >>>>>>>> index e9456e3e74cc..c77a3e631d04 100644
> >>>>>>>> --- a/drivers/tee/optee/smc_abi.c
> >>>>>>>> +++ b/drivers/tee/optee/smc_abi.c
> >>>>>>>> @@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
> >>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
> >>>>>>>>
> >>>>>>>>         platform_set_drvdata(pdev, optee);
> >>>>>>>> -       ctx = teedev_open(optee->teedev);
> >>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
> >>>>>>>>         if (IS_ERR(ctx)) {
> >>>>>>>>                 rc = PTR_ERR(ctx);
> >>>>>>>>                 goto err_supp_uninit;
> >>>>>>>> diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
> >>>>>>>> index 93f3b330aec8..805e1336089d 100644
> >>>>>>>> --- a/drivers/tee/tee_core.c
> >>>>>>>> +++ b/drivers/tee/tee_core.c
> >>>>>>>> @@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> >>>>>>>>                 goto err;
> >>>>>>>>         }
> >>>>>>>>
> >>>>>>>> -       kref_init(&ctx->refcount);
> >>>>>>>>         ctx->teedev = teedev;
> >>>>>>>>         INIT_LIST_HEAD(&ctx->list_shm);
> >>>>>>>>         rc = teedev->desc->ops->open(ctx);
> >>>>>>>> @@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> >>>>>>>>  }
> >>>>>>>>  EXPORT_SYMBOL_GPL(teedev_open);
> >>>>>>>>
> >>>>>>>> -void teedev_ctx_get(struct tee_context *ctx)
> >>>>>>>> +struct tee_context *teedev_get_def_context(struct tee_device *teedev)
> >>>>>>>>  {
> >>>>>>>> -       if (ctx->releasing)
> >>>>>>>> -               return;
> >>>>>>>> +       int rc;
> >>>>>>>> +       struct tee_context *ctx = &teedev->def_ctx;
> >>>>>>>>
> >>>>>>>> -       kref_get(&ctx->refcount);
> >>>>>>>> -}
> >>>>>>>> +       ctx->teedev = teedev;
> >>>>>>>> +       INIT_LIST_HEAD(&ctx->list_shm);
> >>>>>>>> +       rc = teedev->desc->ops->open(ctx);
> >>>>>>>> +       if (rc)
> >>>>>>>> +               return ERR_PTR(rc);
> >>>>>>>
> >>>>>>> I think ctx->teedev and ctx->list_shm must always be initialized or
> >>>>>>> &teedev->def_ctx can't be used in teedev_close_context().
> >>>>>>
> >>>>>> True, but &teedev->def_ctx is never used in teedev_close_context().
> >>>>>> The closing of the &teedev->def_ctx simply ignored. So once opened,
> >>>>>> &teedev->def_ctx will always remain open until the tee_device is alive.
> >>>>>>
> >>>>>>> We could initialize teedev->def_ctx on the first call to teedev_open()
> >>>>>>> on that tee_device. We need a way to tell the
> >>>>>>> teedev->desc->ops->open() to the backed driver that it's initializing
> >>>>>>> the default context though, or optee_open() can't handle the
> >>>>>>> tee-supplicant case properly.
> >>>>>>>
> >>>>>>
> >>>>>> That's a good point. This way, it is guaranteed that there is one def_ctx
> >>>>>> per teedev. There should be a way to tell the open() callback that it is
> >>>>>> a def_ctx, so it is not registered as a supplicant context.
> >>>>>>
> >>>>>>
> >>>>>>> Should we allow this function to be called more than once for each teedev?
> >>>>>>
> >>>>>> Yes, moving to teedev_open() will fix the issue.
> >>>>>>
> >>>>>>> Do we need serialization in this function if it's called after the
> >>>>>>> driver is probed?
> >>>>>>>
> >>>>>>
> >>>>>> True. I'll make sure there is no race.
> >>>>>>
> >>>>>>>>
> >>>>>>>> -static void teedev_ctx_release(struct kref *ref)
> >>>>>>>> -{
> >>>>>>>> -       struct tee_context *ctx = container_of(ref, struct tee_context,
> >>>>>>>> -                                              refcount);
> >>>>>>>> -       ctx->releasing = true;
> >>>>>>>> -       ctx->teedev->desc->ops->release(ctx);
> >>>>>>>> -       kfree(ctx);
> >>>>>>>> +       return ctx;
> >>>>>>>>  }
> >>>>>>>> +EXPORT_SYMBOL_GPL(teedev_get_def_context);
> >>>>>>>>
> >>>>>>>> -void teedev_ctx_put(struct tee_context *ctx)
> >>>>>>>> +void teedev_close_context(struct tee_context *ctx)
> >>>>>>>>  {
> >>>>>>>> -       if (ctx->releasing)
> >>>>>>>> +       struct tee_device *teedev = ctx->teedev;
> >>>>>>>> +       struct tee_shm *shm;
> >>>>>>>> +
> >>>>>>>> +       if (ctx == &teedev->def_ctx)
> >>>>>>>>                 return;
> >>>>>>>>
> >>>>>>>> -       kref_put(&ctx->refcount, teedev_ctx_release);
> >>>>>>>> -}
> >>>>>>>> +       teedev->desc->ops->release(ctx);
> >>>>>>>>
> >>>>>>>> -void teedev_close_context(struct tee_context *ctx)
> >>>>>>>> -{
> >>>>>>>> -       struct tee_device *teedev = ctx->teedev;
> >>>>>>>> +       mutex_lock(&teedev->mutex);
> >>>>>>>> +       list_for_each_entry(shm, &ctx->list_shm, link) {
> >>>>>>>> +               /* Context released. However, shm still holding a teedev reference.
> >>>>>>>> +                * Replace shm->ctx with the default context so that tee_shm_get_from_id()
> >>>>>>>> +                * fails (i.e. it is not accessible from userspace) but shm still
> >>>>>>>> +                * holds a valid context for further clean up, e.g. shm_unregister().
> >>>>>>>> +                */
> >>>>>>>
> >>>>>>> /*
> >>>>>>>  * Please format
> >>>>>>>  * multiline comments
> >>>>>>>  * like this. Please
> >>>>>>>  * keep the lines at
> >>>>>>>  * max 80 columns
> >>>>>>>  * here and at other
> >>>>>>>  * places in the patch-
> >>>>>>>  * set.
> >>>>>>>  */
> >>>>>>>
> >>>>>>
> >>>>>> Ack.
> >>>>>>
> >>>>>>>> +               shm->ctx = &teedev->def_ctx;
> >>>>>>>
> >>>>>>> shm->ctx will always point to a valid context, even if it is the
> >>>>>>> default context. It seems that we can always get hold of the correct
> >>>>>>> teedev via shm->ctx->teedev. Do we need "tee: revert removal of
> >>>>>>> redundant teedev in struct tee_shm"?
> >>>>>>>
> >>>>>>
> >>>>>> It was there in case we wanted to use NULL, but with def_ctx, it is not
> >>>>>> necessary. I am withdrawing that commit. :).
> >>>>>>
> >>>>>>> Shouldn't the shm be removed from the ctx->list_shm and be moved to
> >>>>>>> teedev->def_ctx.list_shm?
> >>>>>
> >>>>> +1
> >>>>>
> >>>>
> >>>> Ack.
> >>>>
> >>>>>>>
> >>>>>>
> >>>>>> Not really. If we put shm in the teedev->def_ctx.list_shm, by the time
> >>>>>> we are closing the def_ctx, the list is guaranteed to be empty.
> >>>>>>
> >>>>>> However, I understand it is cleaner and more consistent to do that rather
> >>>>>> than making changes to tee_shm_put().
> >>>>>>
> >>>>>> I'll do it.
> >>>>>>
> >>>>>>>> +       }
> >>>>>>>> +       mutex_unlock(&teedev->mutex);
> >>>>>>>>
> >>>>>>>> -       teedev_ctx_put(ctx);
> >>>>>>>> +       kfree(ctx);
> >>>>>>>>         tee_device_put(teedev);
> >>>>>>>>  }
> >>>>>>>>  EXPORT_SYMBOL_GPL(teedev_close_context);
> >>>>>>>> @@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
> >>>>>>>>
> >>>>>>>>         teedev->desc = teedesc;
> >>>>>>>>         teedev->pool = pool;
> >>>>>>>> +       /* Only open default context when teedev_get_def_context() called. */
> >>>>>>>> +       teedev->def_ctx.teedev = NULL;
> >>>>>
> >>>>> Why don't you open the device context here only? This will associate
> >>>>> it automatically with teedev lifespan and then
> >>>>> teedev_get_def_context() will just return a reference to that.
> >>>>>
> >>>>> -Sumit
> >>>>>
> >>>>
> >>>> So my assumption is that the tee_devic_alloc() is called as part of
> >>>> the driver initialization; there is no guarantee that at this time the
> >>>> driver is actually ready to accept any open() callback.
> >>>>
> >>>
> >>> The drivers should be able to handle open() callback since we already
> >>> check for !teedesc->ops->open in the beginning of tee_devic_alloc().
> >>> Also, we need to open a device context for !TEE_DESC_PRIVILEGED such
> >>> that we don't open a supplicant device context there.
> >>
> >> It would be nice to have the device context fully initialized when the
> >> probe function returns. How about adding a "bool is_dev_ctx" to struct
> >> tee_context so the open() callback can tell that this is a special
> >> tee_contex?
> >
> > Sure, that will be useful to distinguish the device context from
> > normal client context.
> >
> > -Sumit
> >
>
> So, as far as the open() callback, I do not believe checking if it is not null
> is reasonable for calling it here. Most drivers allocate resources and then
> initialize them. So, assume these steps for a TEE driver:
>  (1) allocate internal data structures,
>  (2) allocate the device,
>  (3) initialize the internal data structurse and then
>  (4) register the device.
>
> Having these steps for a backend driver means that if you call open() at
> step (2), the internal data structures are not ready.
>
> I was originally thinking of going with Jens' suggestion to open dev_ctx in
> the teedev_open(), and use a flag to distinguish the type of context for
> the open() callback
>
> What about this:
> Open the dev_ctx in the tee_device_register(), at the last step before
> setting the TEE_DEVICE_FLAG_REGISTERED flag. Then the open() callback can
> check for this flag to determine if it is a normal context or dev_ctx.
> If the open() is called while the device has not been registered, it should
> handle it differently

That makes sense, the driver should be prepared to handle open() calls
after tee_device_register() anyway.
However, there is no serialization of the flags field in struct
tee_device. Hmm, would it be too hacky for the open() callback to
check if &ctx->teedev.dev_ctx == ctx? We could add a helper function
to wrap that check.

Cheers,
Jens

>
> - Amir
>
> >>
> >> Cheers,
> >> Jens
> >>
> >>>
> >>> -Sumit

Re: [PATCH RFC 3/3] tee: introduce orphan tee_shm and default context

[Sumit Garg]

On Tue, 26 Nov 2024 at 14:03, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>
> On Mon, Nov 25, 2024 at 9:55 PM Amirreza Zarrabi
> <quic_azarrabi@quicinc.com> wrote:
> >
> >
> >
> > On 11/25/2024 6:51 PM, Sumit Garg wrote:
> > > On Mon, 25 Nov 2024 at 12:53, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> > >>
> > >> On Mon, Nov 25, 2024 at 7:14 AM Sumit Garg <sumit.garg@linaro.org> wrote:
> > >>>
> > >>> On Mon, 25 Nov 2024 at 03:00, Amirreza Zarrabi
> > >>> <quic_azarrabi@quicinc.com> wrote:
> > >>>>
> > >>>>
> > >>>> Hi Sumit,
> > >>>>
> > >>>> Thank you so much for the comemnts :).
> > >>>>
> > >>>> On 11/23/2024 9:32 PM, Sumit Garg wrote:
> > >>>>> Hi Amirreza,
> > >>>>>
> > >>>>> Thanks for proposing this.
> > >>>>>
> > >>>>> On Fri, 22 Nov 2024 at 06:38, Amirreza Zarrabi
> > >>>>> <quic_azarrabi@quicinc.com> wrote:
> > >>>>>>
> > >>>>>>
> > >>>>>> On 11/21/2024 11:08 PM, Jens Wiklander wrote:
> > >>>>>>
> > >>>>>> Hi Jens,
> > >>>>>>
> > >>>>>>> Hi Amirreza,
> > >>>>>>>
> > >>>>>>> On Thu, Nov 21, 2024 at 2:37 AM Amirreza Zarrabi
> > >>>>>>> <quic_azarrabi@quicinc.com> wrote:
> > >>>>>>>>
> > >>>>>>>> The default context has a lifespan similar to the tee_device.
> > >>>>>
> > >>>>> Since it's associated with tee_device context, let's call it obvious
> > >>>>> via renaming it as device context instead (s/def_ctx/dev_ctx/ in this
> > >>>>> patch).
> > >>>>>
> > >>>>
> > >>>> Make sense, I'll rename it.
> > >>>>
> > >>>>>>>> It is used as a context for shared memory if the context to which the
> > >>>>>>>> shared memory belongs is released, making the tee_shm an orphan.
> > >>>>>>>> This allows the driver implementing shm_unregister to safely make
> > >>>>>>>> subsequent calls, such as to a supplicant if needed.
> > >>>>>>>>
> > >>>>>>>> It also enables users to free the shared memory while the driver is
> > >>>>>>>> blocked on unregister_tee_device safely.
> > >>>>>>>>
> > >>>>>>>> Preferably, this should be used for all driver internal uses, using
> > >>>>>>>> teedev_get_def_context rather than calling teedev_open.
> > >>>>>
> > >>>>> Makes sense to me.
> > >>>>>
> > >>>>>>>>
> > >>>>>>>> Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
> > >>>>>>>> ---
> > >>>>>>>>  drivers/tee/optee/core.c    |  2 +-
> > >>>>>>>>  drivers/tee/optee/ffa_abi.c |  2 +-
> > >>>>>>>>  drivers/tee/optee/smc_abi.c |  2 +-
> > >>>>>>>>  drivers/tee/tee_core.c      | 83 +++++++++++++++++++++++++++++----------------
> > >>>>>>>>  drivers/tee/tee_private.h   |  3 --
> > >>>>>>>>  drivers/tee/tee_shm.c       | 18 ++--------
> > >>>>>>>>  include/linux/tee_core.h    | 15 ++++++++
> > >>>>>>>>  include/linux/tee_drv.h     |  7 ----
> > >>>>>>>>  8 files changed, 73 insertions(+), 59 deletions(-)
> > >>>>>>>>
> > >>>>>>>> diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
> > >>>>>>>> index c75fddc83576..78d43d0c8014 100644
> > >>>>>>>> --- a/drivers/tee/optee/core.c
> > >>>>>>>> +++ b/drivers/tee/optee/core.c
> > >>>>>>>> @@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
> > >>>>>>>>
> > >>>>>>>>         optee_notif_uninit(optee);
> > >>>>>>>>         optee_shm_arg_cache_uninit(optee);
> > >>>>>>>> -       teedev_close_context(optee->ctx);
> > >>>>>>>> +
> > >>>>>>>>         /*
> > >>>>>>>>          * The two devices have to be unregistered before we can free the
> > >>>>>>>>          * other resources.
> > >>>>>>>> diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
> > >>>>>>>> index f3af5666bb11..6ad94f0788ad 100644
> > >>>>>>>> --- a/drivers/tee/optee/ffa_abi.c
> > >>>>>>>> +++ b/drivers/tee/optee/ffa_abi.c
> > >>>>>>>> @@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
> > >>>>>>>>         optee_shm_arg_cache_init(optee, arg_cache_flags);
> > >>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
> > >>>>>>>>         ffa_dev_set_drvdata(ffa_dev, optee);
> > >>>>>>>> -       ctx = teedev_open(optee->teedev);
> > >>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
> > >>>>>>>>         if (IS_ERR(ctx)) {
> > >>>>>>>>                 rc = PTR_ERR(ctx);
> > >>>>>>>>                 goto err_rhashtable_free;
> > >>>>>>>> diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
> > >>>>>>>> index e9456e3e74cc..c77a3e631d04 100644
> > >>>>>>>> --- a/drivers/tee/optee/smc_abi.c
> > >>>>>>>> +++ b/drivers/tee/optee/smc_abi.c
> > >>>>>>>> @@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
> > >>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
> > >>>>>>>>
> > >>>>>>>>         platform_set_drvdata(pdev, optee);
> > >>>>>>>> -       ctx = teedev_open(optee->teedev);
> > >>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
> > >>>>>>>>         if (IS_ERR(ctx)) {
> > >>>>>>>>                 rc = PTR_ERR(ctx);
> > >>>>>>>>                 goto err_supp_uninit;
> > >>>>>>>> diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
> > >>>>>>>> index 93f3b330aec8..805e1336089d 100644
> > >>>>>>>> --- a/drivers/tee/tee_core.c
> > >>>>>>>> +++ b/drivers/tee/tee_core.c
> > >>>>>>>> @@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> > >>>>>>>>                 goto err;
> > >>>>>>>>         }
> > >>>>>>>>
> > >>>>>>>> -       kref_init(&ctx->refcount);
> > >>>>>>>>         ctx->teedev = teedev;
> > >>>>>>>>         INIT_LIST_HEAD(&ctx->list_shm);
> > >>>>>>>>         rc = teedev->desc->ops->open(ctx);
> > >>>>>>>> @@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> > >>>>>>>>  }
> > >>>>>>>>  EXPORT_SYMBOL_GPL(teedev_open);
> > >>>>>>>>
> > >>>>>>>> -void teedev_ctx_get(struct tee_context *ctx)
> > >>>>>>>> +struct tee_context *teedev_get_def_context(struct tee_device *teedev)
> > >>>>>>>>  {
> > >>>>>>>> -       if (ctx->releasing)
> > >>>>>>>> -               return;
> > >>>>>>>> +       int rc;
> > >>>>>>>> +       struct tee_context *ctx = &teedev->def_ctx;
> > >>>>>>>>
> > >>>>>>>> -       kref_get(&ctx->refcount);
> > >>>>>>>> -}
> > >>>>>>>> +       ctx->teedev = teedev;
> > >>>>>>>> +       INIT_LIST_HEAD(&ctx->list_shm);
> > >>>>>>>> +       rc = teedev->desc->ops->open(ctx);
> > >>>>>>>> +       if (rc)
> > >>>>>>>> +               return ERR_PTR(rc);
> > >>>>>>>
> > >>>>>>> I think ctx->teedev and ctx->list_shm must always be initialized or
> > >>>>>>> &teedev->def_ctx can't be used in teedev_close_context().
> > >>>>>>
> > >>>>>> True, but &teedev->def_ctx is never used in teedev_close_context().
> > >>>>>> The closing of the &teedev->def_ctx simply ignored. So once opened,
> > >>>>>> &teedev->def_ctx will always remain open until the tee_device is alive.
> > >>>>>>
> > >>>>>>> We could initialize teedev->def_ctx on the first call to teedev_open()
> > >>>>>>> on that tee_device. We need a way to tell the
> > >>>>>>> teedev->desc->ops->open() to the backed driver that it's initializing
> > >>>>>>> the default context though, or optee_open() can't handle the
> > >>>>>>> tee-supplicant case properly.
> > >>>>>>>
> > >>>>>>
> > >>>>>> That's a good point. This way, it is guaranteed that there is one def_ctx
> > >>>>>> per teedev. There should be a way to tell the open() callback that it is
> > >>>>>> a def_ctx, so it is not registered as a supplicant context.
> > >>>>>>
> > >>>>>>
> > >>>>>>> Should we allow this function to be called more than once for each teedev?
> > >>>>>>
> > >>>>>> Yes, moving to teedev_open() will fix the issue.
> > >>>>>>
> > >>>>>>> Do we need serialization in this function if it's called after the
> > >>>>>>> driver is probed?
> > >>>>>>>
> > >>>>>>
> > >>>>>> True. I'll make sure there is no race.
> > >>>>>>
> > >>>>>>>>
> > >>>>>>>> -static void teedev_ctx_release(struct kref *ref)
> > >>>>>>>> -{
> > >>>>>>>> -       struct tee_context *ctx = container_of(ref, struct tee_context,
> > >>>>>>>> -                                              refcount);
> > >>>>>>>> -       ctx->releasing = true;
> > >>>>>>>> -       ctx->teedev->desc->ops->release(ctx);
> > >>>>>>>> -       kfree(ctx);
> > >>>>>>>> +       return ctx;
> > >>>>>>>>  }
> > >>>>>>>> +EXPORT_SYMBOL_GPL(teedev_get_def_context);
> > >>>>>>>>
> > >>>>>>>> -void teedev_ctx_put(struct tee_context *ctx)
> > >>>>>>>> +void teedev_close_context(struct tee_context *ctx)
> > >>>>>>>>  {
> > >>>>>>>> -       if (ctx->releasing)
> > >>>>>>>> +       struct tee_device *teedev = ctx->teedev;
> > >>>>>>>> +       struct tee_shm *shm;
> > >>>>>>>> +
> > >>>>>>>> +       if (ctx == &teedev->def_ctx)
> > >>>>>>>>                 return;
> > >>>>>>>>
> > >>>>>>>> -       kref_put(&ctx->refcount, teedev_ctx_release);
> > >>>>>>>> -}
> > >>>>>>>> +       teedev->desc->ops->release(ctx);
> > >>>>>>>>
> > >>>>>>>> -void teedev_close_context(struct tee_context *ctx)
> > >>>>>>>> -{
> > >>>>>>>> -       struct tee_device *teedev = ctx->teedev;
> > >>>>>>>> +       mutex_lock(&teedev->mutex);
> > >>>>>>>> +       list_for_each_entry(shm, &ctx->list_shm, link) {
> > >>>>>>>> +               /* Context released. However, shm still holding a teedev reference.
> > >>>>>>>> +                * Replace shm->ctx with the default context so that tee_shm_get_from_id()
> > >>>>>>>> +                * fails (i.e. it is not accessible from userspace) but shm still
> > >>>>>>>> +                * holds a valid context for further clean up, e.g. shm_unregister().
> > >>>>>>>> +                */
> > >>>>>>>
> > >>>>>>> /*
> > >>>>>>>  * Please format
> > >>>>>>>  * multiline comments
> > >>>>>>>  * like this. Please
> > >>>>>>>  * keep the lines at
> > >>>>>>>  * max 80 columns
> > >>>>>>>  * here and at other
> > >>>>>>>  * places in the patch-
> > >>>>>>>  * set.
> > >>>>>>>  */
> > >>>>>>>
> > >>>>>>
> > >>>>>> Ack.
> > >>>>>>
> > >>>>>>>> +               shm->ctx = &teedev->def_ctx;
> > >>>>>>>
> > >>>>>>> shm->ctx will always point to a valid context, even if it is the
> > >>>>>>> default context. It seems that we can always get hold of the correct
> > >>>>>>> teedev via shm->ctx->teedev. Do we need "tee: revert removal of
> > >>>>>>> redundant teedev in struct tee_shm"?
> > >>>>>>>
> > >>>>>>
> > >>>>>> It was there in case we wanted to use NULL, but with def_ctx, it is not
> > >>>>>> necessary. I am withdrawing that commit. :).
> > >>>>>>
> > >>>>>>> Shouldn't the shm be removed from the ctx->list_shm and be moved to
> > >>>>>>> teedev->def_ctx.list_shm?
> > >>>>>
> > >>>>> +1
> > >>>>>
> > >>>>
> > >>>> Ack.
> > >>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>> Not really. If we put shm in the teedev->def_ctx.list_shm, by the time
> > >>>>>> we are closing the def_ctx, the list is guaranteed to be empty.
> > >>>>>>
> > >>>>>> However, I understand it is cleaner and more consistent to do that rather
> > >>>>>> than making changes to tee_shm_put().
> > >>>>>>
> > >>>>>> I'll do it.
> > >>>>>>
> > >>>>>>>> +       }
> > >>>>>>>> +       mutex_unlock(&teedev->mutex);
> > >>>>>>>>
> > >>>>>>>> -       teedev_ctx_put(ctx);
> > >>>>>>>> +       kfree(ctx);
> > >>>>>>>>         tee_device_put(teedev);
> > >>>>>>>>  }
> > >>>>>>>>  EXPORT_SYMBOL_GPL(teedev_close_context);
> > >>>>>>>> @@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
> > >>>>>>>>
> > >>>>>>>>         teedev->desc = teedesc;
> > >>>>>>>>         teedev->pool = pool;
> > >>>>>>>> +       /* Only open default context when teedev_get_def_context() called. */
> > >>>>>>>> +       teedev->def_ctx.teedev = NULL;
> > >>>>>
> > >>>>> Why don't you open the device context here only? This will associate
> > >>>>> it automatically with teedev lifespan and then
> > >>>>> teedev_get_def_context() will just return a reference to that.
> > >>>>>
> > >>>>> -Sumit
> > >>>>>
> > >>>>
> > >>>> So my assumption is that the tee_devic_alloc() is called as part of
> > >>>> the driver initialization; there is no guarantee that at this time the
> > >>>> driver is actually ready to accept any open() callback.
> > >>>>
> > >>>
> > >>> The drivers should be able to handle open() callback since we already
> > >>> check for !teedesc->ops->open in the beginning of tee_devic_alloc().
> > >>> Also, we need to open a device context for !TEE_DESC_PRIVILEGED such
> > >>> that we don't open a supplicant device context there.
> > >>
> > >> It would be nice to have the device context fully initialized when the
> > >> probe function returns. How about adding a "bool is_dev_ctx" to struct
> > >> tee_context so the open() callback can tell that this is a special
> > >> tee_contex?
> > >
> > > Sure, that will be useful to distinguish the device context from
> > > normal client context.
> > >
> > > -Sumit
> > >
> >
> > So, as far as the open() callback, I do not believe checking if it is not null
> > is reasonable for calling it here. Most drivers allocate resources and then
> > initialize them. So, assume these steps for a TEE driver:
> >  (1) allocate internal data structures,
> >  (2) allocate the device,
> >  (3) initialize the internal data structurse and then
> >  (4) register the device.
> >
> > Having these steps for a backend driver means that if you call open() at
> > step (2), the internal data structures are not ready.

As part of tee_device_alloc(), every driver has to pass "const struct
tee_desc *teedesc" fully initialized. Which internal data structures
are you referring too? Is there any upstream example?

> >
> > I was originally thinking of going with Jens' suggestion to open dev_ctx in
> > the teedev_open(), and use a flag to distinguish the type of context for
> > the open() callback
> >
> > What about this:
> > Open the dev_ctx in the tee_device_register(), at the last step before
> > setting the TEE_DEVICE_FLAG_REGISTERED flag. Then the open() callback can
> > check for this flag to determine if it is a normal context or dev_ctx.
> > If the open() is called while the device has not been registered, it should
> > handle it differently
>
> That makes sense, the driver should be prepared to handle open() calls
> after tee_device_register() anyway.
> However, there is no serialization of the flags field in struct
> tee_device. Hmm, would it be too hacky for the open() callback to
> check if &ctx->teedev.dev_ctx == ctx? We could add a helper function
> to wrap that check.
>

Your suggested change requires every driver to update open() callback
and later other callbacks may have to support it too. IMHO, only
teedev_get_dev_ctx() should be able to return a reference to device
context for usage within the TEE and the implementation driver.

I am still not able to understand why the following won't work with a
clear lifetime for the device context?

tee_device_alloc()
  -> if (!(teedesc->flags & TEE_DESC_PRIVILEGED))
desc->ops->open(&teedev->dev_ctx);

tee_device_put()
  -> if (teedev->dev_ctx) desc->ops->release(&teedev->dev_ctx);

-Sumit

> Cheers,
> Jens
>
> >
> > - Amir
> >
> > >>
> > >> Cheers,
> > >> Jens
> > >>
> > >>>
> > >>> -Sumit

Re: [PATCH RFC 3/3] tee: introduce orphan tee_shm and default context

[Jens Wiklander]

On Tue, Nov 26, 2024 at 1:27 PM Sumit Garg <sumit.garg@linaro.org> wrote:
>
> On Tue, 26 Nov 2024 at 14:03, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > On Mon, Nov 25, 2024 at 9:55 PM Amirreza Zarrabi
> > <quic_azarrabi@quicinc.com> wrote:
> > >
> > >
> > >
> > > On 11/25/2024 6:51 PM, Sumit Garg wrote:
> > > > On Mon, 25 Nov 2024 at 12:53, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> > > >>
> > > >> On Mon, Nov 25, 2024 at 7:14 AM Sumit Garg <sumit.garg@linaro.org> wrote:
> > > >>>
> > > >>> On Mon, 25 Nov 2024 at 03:00, Amirreza Zarrabi
> > > >>> <quic_azarrabi@quicinc.com> wrote:
> > > >>>>
> > > >>>>
> > > >>>> Hi Sumit,
> > > >>>>
> > > >>>> Thank you so much for the comemnts :).
> > > >>>>
> > > >>>> On 11/23/2024 9:32 PM, Sumit Garg wrote:
> > > >>>>> Hi Amirreza,
> > > >>>>>
> > > >>>>> Thanks for proposing this.
> > > >>>>>
> > > >>>>> On Fri, 22 Nov 2024 at 06:38, Amirreza Zarrabi
> > > >>>>> <quic_azarrabi@quicinc.com> wrote:
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On 11/21/2024 11:08 PM, Jens Wiklander wrote:
> > > >>>>>>
> > > >>>>>> Hi Jens,
> > > >>>>>>
> > > >>>>>>> Hi Amirreza,
> > > >>>>>>>
> > > >>>>>>> On Thu, Nov 21, 2024 at 2:37 AM Amirreza Zarrabi
> > > >>>>>>> <quic_azarrabi@quicinc.com> wrote:
> > > >>>>>>>>
> > > >>>>>>>> The default context has a lifespan similar to the tee_device.
> > > >>>>>
> > > >>>>> Since it's associated with tee_device context, let's call it obvious
> > > >>>>> via renaming it as device context instead (s/def_ctx/dev_ctx/ in this
> > > >>>>> patch).
> > > >>>>>
> > > >>>>
> > > >>>> Make sense, I'll rename it.
> > > >>>>
> > > >>>>>>>> It is used as a context for shared memory if the context to which the
> > > >>>>>>>> shared memory belongs is released, making the tee_shm an orphan.
> > > >>>>>>>> This allows the driver implementing shm_unregister to safely make
> > > >>>>>>>> subsequent calls, such as to a supplicant if needed.
> > > >>>>>>>>
> > > >>>>>>>> It also enables users to free the shared memory while the driver is
> > > >>>>>>>> blocked on unregister_tee_device safely.
> > > >>>>>>>>
> > > >>>>>>>> Preferably, this should be used for all driver internal uses, using
> > > >>>>>>>> teedev_get_def_context rather than calling teedev_open.
> > > >>>>>
> > > >>>>> Makes sense to me.
> > > >>>>>
> > > >>>>>>>>
> > > >>>>>>>> Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
> > > >>>>>>>> ---
> > > >>>>>>>>  drivers/tee/optee/core.c    |  2 +-
> > > >>>>>>>>  drivers/tee/optee/ffa_abi.c |  2 +-
> > > >>>>>>>>  drivers/tee/optee/smc_abi.c |  2 +-
> > > >>>>>>>>  drivers/tee/tee_core.c      | 83 +++++++++++++++++++++++++++++----------------
> > > >>>>>>>>  drivers/tee/tee_private.h   |  3 --
> > > >>>>>>>>  drivers/tee/tee_shm.c       | 18 ++--------
> > > >>>>>>>>  include/linux/tee_core.h    | 15 ++++++++
> > > >>>>>>>>  include/linux/tee_drv.h     |  7 ----
> > > >>>>>>>>  8 files changed, 73 insertions(+), 59 deletions(-)
> > > >>>>>>>>
> > > >>>>>>>> diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
> > > >>>>>>>> index c75fddc83576..78d43d0c8014 100644
> > > >>>>>>>> --- a/drivers/tee/optee/core.c
> > > >>>>>>>> +++ b/drivers/tee/optee/core.c
> > > >>>>>>>> @@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
> > > >>>>>>>>
> > > >>>>>>>>         optee_notif_uninit(optee);
> > > >>>>>>>>         optee_shm_arg_cache_uninit(optee);
> > > >>>>>>>> -       teedev_close_context(optee->ctx);
> > > >>>>>>>> +
> > > >>>>>>>>         /*
> > > >>>>>>>>          * The two devices have to be unregistered before we can free the
> > > >>>>>>>>          * other resources.
> > > >>>>>>>> diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
> > > >>>>>>>> index f3af5666bb11..6ad94f0788ad 100644
> > > >>>>>>>> --- a/drivers/tee/optee/ffa_abi.c
> > > >>>>>>>> +++ b/drivers/tee/optee/ffa_abi.c
> > > >>>>>>>> @@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
> > > >>>>>>>>         optee_shm_arg_cache_init(optee, arg_cache_flags);
> > > >>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
> > > >>>>>>>>         ffa_dev_set_drvdata(ffa_dev, optee);
> > > >>>>>>>> -       ctx = teedev_open(optee->teedev);
> > > >>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
> > > >>>>>>>>         if (IS_ERR(ctx)) {
> > > >>>>>>>>                 rc = PTR_ERR(ctx);
> > > >>>>>>>>                 goto err_rhashtable_free;
> > > >>>>>>>> diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
> > > >>>>>>>> index e9456e3e74cc..c77a3e631d04 100644
> > > >>>>>>>> --- a/drivers/tee/optee/smc_abi.c
> > > >>>>>>>> +++ b/drivers/tee/optee/smc_abi.c
> > > >>>>>>>> @@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
> > > >>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
> > > >>>>>>>>
> > > >>>>>>>>         platform_set_drvdata(pdev, optee);
> > > >>>>>>>> -       ctx = teedev_open(optee->teedev);
> > > >>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
> > > >>>>>>>>         if (IS_ERR(ctx)) {
> > > >>>>>>>>                 rc = PTR_ERR(ctx);
> > > >>>>>>>>                 goto err_supp_uninit;
> > > >>>>>>>> diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
> > > >>>>>>>> index 93f3b330aec8..805e1336089d 100644
> > > >>>>>>>> --- a/drivers/tee/tee_core.c
> > > >>>>>>>> +++ b/drivers/tee/tee_core.c
> > > >>>>>>>> @@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> > > >>>>>>>>                 goto err;
> > > >>>>>>>>         }
> > > >>>>>>>>
> > > >>>>>>>> -       kref_init(&ctx->refcount);
> > > >>>>>>>>         ctx->teedev = teedev;
> > > >>>>>>>>         INIT_LIST_HEAD(&ctx->list_shm);
> > > >>>>>>>>         rc = teedev->desc->ops->open(ctx);
> > > >>>>>>>> @@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> > > >>>>>>>>  }
> > > >>>>>>>>  EXPORT_SYMBOL_GPL(teedev_open);
> > > >>>>>>>>
> > > >>>>>>>> -void teedev_ctx_get(struct tee_context *ctx)
> > > >>>>>>>> +struct tee_context *teedev_get_def_context(struct tee_device *teedev)
> > > >>>>>>>>  {
> > > >>>>>>>> -       if (ctx->releasing)
> > > >>>>>>>> -               return;
> > > >>>>>>>> +       int rc;
> > > >>>>>>>> +       struct tee_context *ctx = &teedev->def_ctx;
> > > >>>>>>>>
> > > >>>>>>>> -       kref_get(&ctx->refcount);
> > > >>>>>>>> -}
> > > >>>>>>>> +       ctx->teedev = teedev;
> > > >>>>>>>> +       INIT_LIST_HEAD(&ctx->list_shm);
> > > >>>>>>>> +       rc = teedev->desc->ops->open(ctx);
> > > >>>>>>>> +       if (rc)
> > > >>>>>>>> +               return ERR_PTR(rc);
> > > >>>>>>>
> > > >>>>>>> I think ctx->teedev and ctx->list_shm must always be initialized or
> > > >>>>>>> &teedev->def_ctx can't be used in teedev_close_context().
> > > >>>>>>
> > > >>>>>> True, but &teedev->def_ctx is never used in teedev_close_context().
> > > >>>>>> The closing of the &teedev->def_ctx simply ignored. So once opened,
> > > >>>>>> &teedev->def_ctx will always remain open until the tee_device is alive.
> > > >>>>>>
> > > >>>>>>> We could initialize teedev->def_ctx on the first call to teedev_open()
> > > >>>>>>> on that tee_device. We need a way to tell the
> > > >>>>>>> teedev->desc->ops->open() to the backed driver that it's initializing
> > > >>>>>>> the default context though, or optee_open() can't handle the
> > > >>>>>>> tee-supplicant case properly.
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>> That's a good point. This way, it is guaranteed that there is one def_ctx
> > > >>>>>> per teedev. There should be a way to tell the open() callback that it is
> > > >>>>>> a def_ctx, so it is not registered as a supplicant context.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>> Should we allow this function to be called more than once for each teedev?
> > > >>>>>>
> > > >>>>>> Yes, moving to teedev_open() will fix the issue.
> > > >>>>>>
> > > >>>>>>> Do we need serialization in this function if it's called after the
> > > >>>>>>> driver is probed?
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>> True. I'll make sure there is no race.
> > > >>>>>>
> > > >>>>>>>>
> > > >>>>>>>> -static void teedev_ctx_release(struct kref *ref)
> > > >>>>>>>> -{
> > > >>>>>>>> -       struct tee_context *ctx = container_of(ref, struct tee_context,
> > > >>>>>>>> -                                              refcount);
> > > >>>>>>>> -       ctx->releasing = true;
> > > >>>>>>>> -       ctx->teedev->desc->ops->release(ctx);
> > > >>>>>>>> -       kfree(ctx);
> > > >>>>>>>> +       return ctx;
> > > >>>>>>>>  }
> > > >>>>>>>> +EXPORT_SYMBOL_GPL(teedev_get_def_context);
> > > >>>>>>>>
> > > >>>>>>>> -void teedev_ctx_put(struct tee_context *ctx)
> > > >>>>>>>> +void teedev_close_context(struct tee_context *ctx)
> > > >>>>>>>>  {
> > > >>>>>>>> -       if (ctx->releasing)
> > > >>>>>>>> +       struct tee_device *teedev = ctx->teedev;
> > > >>>>>>>> +       struct tee_shm *shm;
> > > >>>>>>>> +
> > > >>>>>>>> +       if (ctx == &teedev->def_ctx)
> > > >>>>>>>>                 return;
> > > >>>>>>>>
> > > >>>>>>>> -       kref_put(&ctx->refcount, teedev_ctx_release);
> > > >>>>>>>> -}
> > > >>>>>>>> +       teedev->desc->ops->release(ctx);
> > > >>>>>>>>
> > > >>>>>>>> -void teedev_close_context(struct tee_context *ctx)
> > > >>>>>>>> -{
> > > >>>>>>>> -       struct tee_device *teedev = ctx->teedev;
> > > >>>>>>>> +       mutex_lock(&teedev->mutex);
> > > >>>>>>>> +       list_for_each_entry(shm, &ctx->list_shm, link) {
> > > >>>>>>>> +               /* Context released. However, shm still holding a teedev reference.
> > > >>>>>>>> +                * Replace shm->ctx with the default context so that tee_shm_get_from_id()
> > > >>>>>>>> +                * fails (i.e. it is not accessible from userspace) but shm still
> > > >>>>>>>> +                * holds a valid context for further clean up, e.g. shm_unregister().
> > > >>>>>>>> +                */
> > > >>>>>>>
> > > >>>>>>> /*
> > > >>>>>>>  * Please format
> > > >>>>>>>  * multiline comments
> > > >>>>>>>  * like this. Please
> > > >>>>>>>  * keep the lines at
> > > >>>>>>>  * max 80 columns
> > > >>>>>>>  * here and at other
> > > >>>>>>>  * places in the patch-
> > > >>>>>>>  * set.
> > > >>>>>>>  */
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>> Ack.
> > > >>>>>>
> > > >>>>>>>> +               shm->ctx = &teedev->def_ctx;
> > > >>>>>>>
> > > >>>>>>> shm->ctx will always point to a valid context, even if it is the
> > > >>>>>>> default context. It seems that we can always get hold of the correct
> > > >>>>>>> teedev via shm->ctx->teedev. Do we need "tee: revert removal of
> > > >>>>>>> redundant teedev in struct tee_shm"?
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>> It was there in case we wanted to use NULL, but with def_ctx, it is not
> > > >>>>>> necessary. I am withdrawing that commit. :).
> > > >>>>>>
> > > >>>>>>> Shouldn't the shm be removed from the ctx->list_shm and be moved to
> > > >>>>>>> teedev->def_ctx.list_shm?
> > > >>>>>
> > > >>>>> +1
> > > >>>>>
> > > >>>>
> > > >>>> Ack.
> > > >>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>> Not really. If we put shm in the teedev->def_ctx.list_shm, by the time
> > > >>>>>> we are closing the def_ctx, the list is guaranteed to be empty.
> > > >>>>>>
> > > >>>>>> However, I understand it is cleaner and more consistent to do that rather
> > > >>>>>> than making changes to tee_shm_put().
> > > >>>>>>
> > > >>>>>> I'll do it.
> > > >>>>>>
> > > >>>>>>>> +       }
> > > >>>>>>>> +       mutex_unlock(&teedev->mutex);
> > > >>>>>>>>
> > > >>>>>>>> -       teedev_ctx_put(ctx);
> > > >>>>>>>> +       kfree(ctx);
> > > >>>>>>>>         tee_device_put(teedev);
> > > >>>>>>>>  }
> > > >>>>>>>>  EXPORT_SYMBOL_GPL(teedev_close_context);
> > > >>>>>>>> @@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
> > > >>>>>>>>
> > > >>>>>>>>         teedev->desc = teedesc;
> > > >>>>>>>>         teedev->pool = pool;
> > > >>>>>>>> +       /* Only open default context when teedev_get_def_context() called. */
> > > >>>>>>>> +       teedev->def_ctx.teedev = NULL;
> > > >>>>>
> > > >>>>> Why don't you open the device context here only? This will associate
> > > >>>>> it automatically with teedev lifespan and then
> > > >>>>> teedev_get_def_context() will just return a reference to that.
> > > >>>>>
> > > >>>>> -Sumit
> > > >>>>>
> > > >>>>
> > > >>>> So my assumption is that the tee_devic_alloc() is called as part of
> > > >>>> the driver initialization; there is no guarantee that at this time the
> > > >>>> driver is actually ready to accept any open() callback.
> > > >>>>
> > > >>>
> > > >>> The drivers should be able to handle open() callback since we already
> > > >>> check for !teedesc->ops->open in the beginning of tee_devic_alloc().
> > > >>> Also, we need to open a device context for !TEE_DESC_PRIVILEGED such
> > > >>> that we don't open a supplicant device context there.
> > > >>
> > > >> It would be nice to have the device context fully initialized when the
> > > >> probe function returns. How about adding a "bool is_dev_ctx" to struct
> > > >> tee_context so the open() callback can tell that this is a special
> > > >> tee_contex?
> > > >
> > > > Sure, that will be useful to distinguish the device context from
> > > > normal client context.
> > > >
> > > > -Sumit
> > > >
> > >
> > > So, as far as the open() callback, I do not believe checking if it is not null
> > > is reasonable for calling it here. Most drivers allocate resources and then
> > > initialize them. So, assume these steps for a TEE driver:
> > >  (1) allocate internal data structures,
> > >  (2) allocate the device,
> > >  (3) initialize the internal data structurse and then
> > >  (4) register the device.
> > >
> > > Having these steps for a backend driver means that if you call open() at
> > > step (2), the internal data structures are not ready.
>
> As part of tee_device_alloc(), every driver has to pass "const struct
> tee_desc *teedesc" fully initialized. Which internal data structures
> are you referring too? Is there any upstream example?

It's reasonable to wait with the open() callback until step 4 above,
which should correspond with the tee_device_register() call. Data
written only once doesn't need serialized access if the fields are
only accessed after they have been fully initialized.

>
> > >
> > > I was originally thinking of going with Jens' suggestion to open dev_ctx in
> > > the teedev_open(), and use a flag to distinguish the type of context for
> > > the open() callback
> > >
> > > What about this:
> > > Open the dev_ctx in the tee_device_register(), at the last step before
> > > setting the TEE_DEVICE_FLAG_REGISTERED flag. Then the open() callback can
> > > check for this flag to determine if it is a normal context or dev_ctx.
> > > If the open() is called while the device has not been registered, it should
> > > handle it differently
> >
> > That makes sense, the driver should be prepared to handle open() calls
> > after tee_device_register() anyway.
> > However, there is no serialization of the flags field in struct
> > tee_device. Hmm, would it be too hacky for the open() callback to
> > check if &ctx->teedev.dev_ctx == ctx? We could add a helper function
> > to wrap that check.
> >
>
> Your suggested change requires every driver to update open() callback
> and later other callbacks may have to support it too. IMHO, only
> teedev_get_dev_ctx() should be able to return a reference to device
> context for usage within the TEE and the implementation driver.

Yes, but it's only the OP-TEE driver that needs anything special. It
looks like the others can be left unchanged.

>
> I am still not able to understand why the following won't work with a
> clear lifetime for the device context?
>
> tee_device_alloc()
>   -> if (!(teedesc->flags & TEE_DESC_PRIVILEGED))
> desc->ops->open(&teedev->dev_ctx);

We must also have a fully initialized dev_ctx for the supplicant
device. I'd rather delay the open() callback until
tee_device_register() since the dev_ctx is guaranteed not to be needed
before that.

>
> tee_device_put()
>   -> if (teedev->dev_ctx) desc->ops->release(&teedev->dev_ctx);

teedev->dev_ctx is supposed to be embedded in struct tee_device, so
the if isn't needed.

Cheers,
Jens

>
> -Sumit
>
> > Cheers,
> > Jens
> >
> > >
> > > - Amir
> > >
> > > >>
> > > >> Cheers,
> > > >> Jens
> > > >>
> > > >>>
> > > >>> -Sumit

Re: [PATCH RFC 3/3] tee: introduce orphan tee_shm and default context

[Sumit Garg]

On Tue, 26 Nov 2024 at 20:52, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>
> On Tue, Nov 26, 2024 at 1:27 PM Sumit Garg <sumit.garg@linaro.org> wrote:
> >
> > On Tue, 26 Nov 2024 at 14:03, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> > >
> > > On Mon, Nov 25, 2024 at 9:55 PM Amirreza Zarrabi
> > > <quic_azarrabi@quicinc.com> wrote:
> > > >
> > > >
> > > >
> > > > On 11/25/2024 6:51 PM, Sumit Garg wrote:
> > > > > On Mon, 25 Nov 2024 at 12:53, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> > > > >>
> > > > >> On Mon, Nov 25, 2024 at 7:14 AM Sumit Garg <sumit.garg@linaro.org> wrote:
> > > > >>>
> > > > >>> On Mon, 25 Nov 2024 at 03:00, Amirreza Zarrabi
> > > > >>> <quic_azarrabi@quicinc.com> wrote:
> > > > >>>>
> > > > >>>>
> > > > >>>> Hi Sumit,
> > > > >>>>
> > > > >>>> Thank you so much for the comemnts :).
> > > > >>>>
> > > > >>>> On 11/23/2024 9:32 PM, Sumit Garg wrote:
> > > > >>>>> Hi Amirreza,
> > > > >>>>>
> > > > >>>>> Thanks for proposing this.
> > > > >>>>>
> > > > >>>>> On Fri, 22 Nov 2024 at 06:38, Amirreza Zarrabi
> > > > >>>>> <quic_azarrabi@quicinc.com> wrote:
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> On 11/21/2024 11:08 PM, Jens Wiklander wrote:
> > > > >>>>>>
> > > > >>>>>> Hi Jens,
> > > > >>>>>>
> > > > >>>>>>> Hi Amirreza,
> > > > >>>>>>>
> > > > >>>>>>> On Thu, Nov 21, 2024 at 2:37 AM Amirreza Zarrabi
> > > > >>>>>>> <quic_azarrabi@quicinc.com> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>> The default context has a lifespan similar to the tee_device.
> > > > >>>>>
> > > > >>>>> Since it's associated with tee_device context, let's call it obvious
> > > > >>>>> via renaming it as device context instead (s/def_ctx/dev_ctx/ in this
> > > > >>>>> patch).
> > > > >>>>>
> > > > >>>>
> > > > >>>> Make sense, I'll rename it.
> > > > >>>>
> > > > >>>>>>>> It is used as a context for shared memory if the context to which the
> > > > >>>>>>>> shared memory belongs is released, making the tee_shm an orphan.
> > > > >>>>>>>> This allows the driver implementing shm_unregister to safely make
> > > > >>>>>>>> subsequent calls, such as to a supplicant if needed.
> > > > >>>>>>>>
> > > > >>>>>>>> It also enables users to free the shared memory while the driver is
> > > > >>>>>>>> blocked on unregister_tee_device safely.
> > > > >>>>>>>>
> > > > >>>>>>>> Preferably, this should be used for all driver internal uses, using
> > > > >>>>>>>> teedev_get_def_context rather than calling teedev_open.
> > > > >>>>>
> > > > >>>>> Makes sense to me.
> > > > >>>>>
> > > > >>>>>>>>
> > > > >>>>>>>> Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
> > > > >>>>>>>> ---
> > > > >>>>>>>>  drivers/tee/optee/core.c    |  2 +-
> > > > >>>>>>>>  drivers/tee/optee/ffa_abi.c |  2 +-
> > > > >>>>>>>>  drivers/tee/optee/smc_abi.c |  2 +-
> > > > >>>>>>>>  drivers/tee/tee_core.c      | 83 +++++++++++++++++++++++++++++----------------
> > > > >>>>>>>>  drivers/tee/tee_private.h   |  3 --
> > > > >>>>>>>>  drivers/tee/tee_shm.c       | 18 ++--------
> > > > >>>>>>>>  include/linux/tee_core.h    | 15 ++++++++
> > > > >>>>>>>>  include/linux/tee_drv.h     |  7 ----
> > > > >>>>>>>>  8 files changed, 73 insertions(+), 59 deletions(-)
> > > > >>>>>>>>
> > > > >>>>>>>> diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
> > > > >>>>>>>> index c75fddc83576..78d43d0c8014 100644
> > > > >>>>>>>> --- a/drivers/tee/optee/core.c
> > > > >>>>>>>> +++ b/drivers/tee/optee/core.c
> > > > >>>>>>>> @@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
> > > > >>>>>>>>
> > > > >>>>>>>>         optee_notif_uninit(optee);
> > > > >>>>>>>>         optee_shm_arg_cache_uninit(optee);
> > > > >>>>>>>> -       teedev_close_context(optee->ctx);
> > > > >>>>>>>> +
> > > > >>>>>>>>         /*
> > > > >>>>>>>>          * The two devices have to be unregistered before we can free the
> > > > >>>>>>>>          * other resources.
> > > > >>>>>>>> diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
> > > > >>>>>>>> index f3af5666bb11..6ad94f0788ad 100644
> > > > >>>>>>>> --- a/drivers/tee/optee/ffa_abi.c
> > > > >>>>>>>> +++ b/drivers/tee/optee/ffa_abi.c
> > > > >>>>>>>> @@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
> > > > >>>>>>>>         optee_shm_arg_cache_init(optee, arg_cache_flags);
> > > > >>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
> > > > >>>>>>>>         ffa_dev_set_drvdata(ffa_dev, optee);
> > > > >>>>>>>> -       ctx = teedev_open(optee->teedev);
> > > > >>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
> > > > >>>>>>>>         if (IS_ERR(ctx)) {
> > > > >>>>>>>>                 rc = PTR_ERR(ctx);
> > > > >>>>>>>>                 goto err_rhashtable_free;
> > > > >>>>>>>> diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
> > > > >>>>>>>> index e9456e3e74cc..c77a3e631d04 100644
> > > > >>>>>>>> --- a/drivers/tee/optee/smc_abi.c
> > > > >>>>>>>> +++ b/drivers/tee/optee/smc_abi.c
> > > > >>>>>>>> @@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
> > > > >>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
> > > > >>>>>>>>
> > > > >>>>>>>>         platform_set_drvdata(pdev, optee);
> > > > >>>>>>>> -       ctx = teedev_open(optee->teedev);
> > > > >>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
> > > > >>>>>>>>         if (IS_ERR(ctx)) {
> > > > >>>>>>>>                 rc = PTR_ERR(ctx);
> > > > >>>>>>>>                 goto err_supp_uninit;
> > > > >>>>>>>> diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
> > > > >>>>>>>> index 93f3b330aec8..805e1336089d 100644
> > > > >>>>>>>> --- a/drivers/tee/tee_core.c
> > > > >>>>>>>> +++ b/drivers/tee/tee_core.c
> > > > >>>>>>>> @@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> > > > >>>>>>>>                 goto err;
> > > > >>>>>>>>         }
> > > > >>>>>>>>
> > > > >>>>>>>> -       kref_init(&ctx->refcount);
> > > > >>>>>>>>         ctx->teedev = teedev;
> > > > >>>>>>>>         INIT_LIST_HEAD(&ctx->list_shm);
> > > > >>>>>>>>         rc = teedev->desc->ops->open(ctx);
> > > > >>>>>>>> @@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> > > > >>>>>>>>  }
> > > > >>>>>>>>  EXPORT_SYMBOL_GPL(teedev_open);
> > > > >>>>>>>>
> > > > >>>>>>>> -void teedev_ctx_get(struct tee_context *ctx)
> > > > >>>>>>>> +struct tee_context *teedev_get_def_context(struct tee_device *teedev)
> > > > >>>>>>>>  {
> > > > >>>>>>>> -       if (ctx->releasing)
> > > > >>>>>>>> -               return;
> > > > >>>>>>>> +       int rc;
> > > > >>>>>>>> +       struct tee_context *ctx = &teedev->def_ctx;
> > > > >>>>>>>>
> > > > >>>>>>>> -       kref_get(&ctx->refcount);
> > > > >>>>>>>> -}
> > > > >>>>>>>> +       ctx->teedev = teedev;
> > > > >>>>>>>> +       INIT_LIST_HEAD(&ctx->list_shm);
> > > > >>>>>>>> +       rc = teedev->desc->ops->open(ctx);
> > > > >>>>>>>> +       if (rc)
> > > > >>>>>>>> +               return ERR_PTR(rc);
> > > > >>>>>>>
> > > > >>>>>>> I think ctx->teedev and ctx->list_shm must always be initialized or
> > > > >>>>>>> &teedev->def_ctx can't be used in teedev_close_context().
> > > > >>>>>>
> > > > >>>>>> True, but &teedev->def_ctx is never used in teedev_close_context().
> > > > >>>>>> The closing of the &teedev->def_ctx simply ignored. So once opened,
> > > > >>>>>> &teedev->def_ctx will always remain open until the tee_device is alive.
> > > > >>>>>>
> > > > >>>>>>> We could initialize teedev->def_ctx on the first call to teedev_open()
> > > > >>>>>>> on that tee_device. We need a way to tell the
> > > > >>>>>>> teedev->desc->ops->open() to the backed driver that it's initializing
> > > > >>>>>>> the default context though, or optee_open() can't handle the
> > > > >>>>>>> tee-supplicant case properly.
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>> That's a good point. This way, it is guaranteed that there is one def_ctx
> > > > >>>>>> per teedev. There should be a way to tell the open() callback that it is
> > > > >>>>>> a def_ctx, so it is not registered as a supplicant context.
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>> Should we allow this function to be called more than once for each teedev?
> > > > >>>>>>
> > > > >>>>>> Yes, moving to teedev_open() will fix the issue.
> > > > >>>>>>
> > > > >>>>>>> Do we need serialization in this function if it's called after the
> > > > >>>>>>> driver is probed?
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>> True. I'll make sure there is no race.
> > > > >>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>> -static void teedev_ctx_release(struct kref *ref)
> > > > >>>>>>>> -{
> > > > >>>>>>>> -       struct tee_context *ctx = container_of(ref, struct tee_context,
> > > > >>>>>>>> -                                              refcount);
> > > > >>>>>>>> -       ctx->releasing = true;
> > > > >>>>>>>> -       ctx->teedev->desc->ops->release(ctx);
> > > > >>>>>>>> -       kfree(ctx);
> > > > >>>>>>>> +       return ctx;
> > > > >>>>>>>>  }
> > > > >>>>>>>> +EXPORT_SYMBOL_GPL(teedev_get_def_context);
> > > > >>>>>>>>
> > > > >>>>>>>> -void teedev_ctx_put(struct tee_context *ctx)
> > > > >>>>>>>> +void teedev_close_context(struct tee_context *ctx)
> > > > >>>>>>>>  {
> > > > >>>>>>>> -       if (ctx->releasing)
> > > > >>>>>>>> +       struct tee_device *teedev = ctx->teedev;
> > > > >>>>>>>> +       struct tee_shm *shm;
> > > > >>>>>>>> +
> > > > >>>>>>>> +       if (ctx == &teedev->def_ctx)
> > > > >>>>>>>>                 return;
> > > > >>>>>>>>
> > > > >>>>>>>> -       kref_put(&ctx->refcount, teedev_ctx_release);
> > > > >>>>>>>> -}
> > > > >>>>>>>> +       teedev->desc->ops->release(ctx);
> > > > >>>>>>>>
> > > > >>>>>>>> -void teedev_close_context(struct tee_context *ctx)
> > > > >>>>>>>> -{
> > > > >>>>>>>> -       struct tee_device *teedev = ctx->teedev;
> > > > >>>>>>>> +       mutex_lock(&teedev->mutex);
> > > > >>>>>>>> +       list_for_each_entry(shm, &ctx->list_shm, link) {
> > > > >>>>>>>> +               /* Context released. However, shm still holding a teedev reference.
> > > > >>>>>>>> +                * Replace shm->ctx with the default context so that tee_shm_get_from_id()
> > > > >>>>>>>> +                * fails (i.e. it is not accessible from userspace) but shm still
> > > > >>>>>>>> +                * holds a valid context for further clean up, e.g. shm_unregister().
> > > > >>>>>>>> +                */
> > > > >>>>>>>
> > > > >>>>>>> /*
> > > > >>>>>>>  * Please format
> > > > >>>>>>>  * multiline comments
> > > > >>>>>>>  * like this. Please
> > > > >>>>>>>  * keep the lines at
> > > > >>>>>>>  * max 80 columns
> > > > >>>>>>>  * here and at other
> > > > >>>>>>>  * places in the patch-
> > > > >>>>>>>  * set.
> > > > >>>>>>>  */
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>> Ack.
> > > > >>>>>>
> > > > >>>>>>>> +               shm->ctx = &teedev->def_ctx;
> > > > >>>>>>>
> > > > >>>>>>> shm->ctx will always point to a valid context, even if it is the
> > > > >>>>>>> default context. It seems that we can always get hold of the correct
> > > > >>>>>>> teedev via shm->ctx->teedev. Do we need "tee: revert removal of
> > > > >>>>>>> redundant teedev in struct tee_shm"?
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>> It was there in case we wanted to use NULL, but with def_ctx, it is not
> > > > >>>>>> necessary. I am withdrawing that commit. :).
> > > > >>>>>>
> > > > >>>>>>> Shouldn't the shm be removed from the ctx->list_shm and be moved to
> > > > >>>>>>> teedev->def_ctx.list_shm?
> > > > >>>>>
> > > > >>>>> +1
> > > > >>>>>
> > > > >>>>
> > > > >>>> Ack.
> > > > >>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>> Not really. If we put shm in the teedev->def_ctx.list_shm, by the time
> > > > >>>>>> we are closing the def_ctx, the list is guaranteed to be empty.
> > > > >>>>>>
> > > > >>>>>> However, I understand it is cleaner and more consistent to do that rather
> > > > >>>>>> than making changes to tee_shm_put().
> > > > >>>>>>
> > > > >>>>>> I'll do it.
> > > > >>>>>>
> > > > >>>>>>>> +       }
> > > > >>>>>>>> +       mutex_unlock(&teedev->mutex);
> > > > >>>>>>>>
> > > > >>>>>>>> -       teedev_ctx_put(ctx);
> > > > >>>>>>>> +       kfree(ctx);
> > > > >>>>>>>>         tee_device_put(teedev);
> > > > >>>>>>>>  }
> > > > >>>>>>>>  EXPORT_SYMBOL_GPL(teedev_close_context);
> > > > >>>>>>>> @@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
> > > > >>>>>>>>
> > > > >>>>>>>>         teedev->desc = teedesc;
> > > > >>>>>>>>         teedev->pool = pool;
> > > > >>>>>>>> +       /* Only open default context when teedev_get_def_context() called. */
> > > > >>>>>>>> +       teedev->def_ctx.teedev = NULL;
> > > > >>>>>
> > > > >>>>> Why don't you open the device context here only? This will associate
> > > > >>>>> it automatically with teedev lifespan and then
> > > > >>>>> teedev_get_def_context() will just return a reference to that.
> > > > >>>>>
> > > > >>>>> -Sumit
> > > > >>>>>
> > > > >>>>
> > > > >>>> So my assumption is that the tee_devic_alloc() is called as part of
> > > > >>>> the driver initialization; there is no guarantee that at this time the
> > > > >>>> driver is actually ready to accept any open() callback.
> > > > >>>>
> > > > >>>
> > > > >>> The drivers should be able to handle open() callback since we already
> > > > >>> check for !teedesc->ops->open in the beginning of tee_devic_alloc().
> > > > >>> Also, we need to open a device context for !TEE_DESC_PRIVILEGED such
> > > > >>> that we don't open a supplicant device context there.
> > > > >>
> > > > >> It would be nice to have the device context fully initialized when the
> > > > >> probe function returns. How about adding a "bool is_dev_ctx" to struct
> > > > >> tee_context so the open() callback can tell that this is a special
> > > > >> tee_contex?
> > > > >
> > > > > Sure, that will be useful to distinguish the device context from
> > > > > normal client context.
> > > > >
> > > > > -Sumit
> > > > >
> > > >
> > > > So, as far as the open() callback, I do not believe checking if it is not null
> > > > is reasonable for calling it here. Most drivers allocate resources and then
> > > > initialize them. So, assume these steps for a TEE driver:
> > > >  (1) allocate internal data structures,
> > > >  (2) allocate the device,
> > > >  (3) initialize the internal data structurse and then
> > > >  (4) register the device.
> > > >
> > > > Having these steps for a backend driver means that if you call open() at
> > > > step (2), the internal data structures are not ready.
> >
> > As part of tee_device_alloc(), every driver has to pass "const struct
> > tee_desc *teedesc" fully initialized. Which internal data structures
> > are you referring too? Is there any upstream example?
>
> It's reasonable to wait with the open() callback until step 4 above,
> which should correspond with the tee_device_register() call. Data
> written only once doesn't need serialized access if the fields are
> only accessed after they have been fully initialized.

Fair enough, I can live with the device context opened after registering it.

>
> >
> > > >
> > > > I was originally thinking of going with Jens' suggestion to open dev_ctx in
> > > > the teedev_open(), and use a flag to distinguish the type of context for
> > > > the open() callback
> > > >
> > > > What about this:
> > > > Open the dev_ctx in the tee_device_register(), at the last step before
> > > > setting the TEE_DEVICE_FLAG_REGISTERED flag. Then the open() callback can
> > > > check for this flag to determine if it is a normal context or dev_ctx.
> > > > If the open() is called while the device has not been registered, it should
> > > > handle it differently
> > >
> > > That makes sense, the driver should be prepared to handle open() calls
> > > after tee_device_register() anyway.
> > > However, there is no serialization of the flags field in struct
> > > tee_device. Hmm, would it be too hacky for the open() callback to
> > > check if &ctx->teedev.dev_ctx == ctx? We could add a helper function
> > > to wrap that check.
> > >
> >
> > Your suggested change requires every driver to update open() callback
> > and later other callbacks may have to support it too. IMHO, only
> > teedev_get_dev_ctx() should be able to return a reference to device
> > context for usage within the TEE and the implementation driver.
>
> Yes, but it's only the OP-TEE driver that needs anything special. It
> looks like the others can be left unchanged.

I suppose it's most likely the upcoming QTEE driver requiring it.

>
> >
> > I am still not able to understand why the following won't work with a
> > clear lifetime for the device context?
> >
> > tee_device_alloc()
> >   -> if (!(teedesc->flags & TEE_DESC_PRIVILEGED))
> > desc->ops->open(&teedev->dev_ctx);
>
> We must also have a fully initialized dev_ctx for the supplicant
> device.

Currently I only see following for OP-TEE driver:

ctx = teedev_open(optee->teedev);

And I can't see anything like below:

ctx = teedev_open(optee->supp_teedev);

Where do you think that the dev_ctx is required for a supplicant
device? AFAICS, currently opening a context with the supplicant device
means that the supplicant daemon is available to handle RPCs which
won't be possible during OP-TEE driver probe. Am I missing something?

> I'd rather delay the open() callback until
> tee_device_register() since the dev_ctx is guaranteed not to be needed
> before that.

Okay, the updated call chain can look like:

tee_device_register()
  -> if (!(teedev->desc->flags & TEE_DESC_PRIVILEGED))
         desc->ops->open(&teedev->dev_ctx);
>
> >
> > tee_device_put()
> >   -> if (teedev->dev_ctx) desc->ops->release(&teedev->dev_ctx);
>
> teedev->dev_ctx is supposed to be embedded in struct tee_device, so
> the if isn't needed.

I added "if" to cover the case when dev_ctx is not initialized for the
supplicant device.

-Sumit

>
> Cheers,
> Jens
>
> >
> > -Sumit
> >
> > > Cheers,
> > > Jens
> > >
> > > >
> > > > - Amir
> > > >
> > > > >>
> > > > >> Cheers,
> > > > >> Jens
> > > > >>
> > > > >>>
> > > > >>> -Sumit

Re: [PATCH RFC 3/3] tee: introduce orphan tee_shm and default context

[Jens Wiklander]

On Wed, Nov 27, 2024 at 7:02 AM Sumit Garg <sumit.garg@linaro.org> wrote:
>
> On Tue, 26 Nov 2024 at 20:52, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >
> > On Tue, Nov 26, 2024 at 1:27 PM Sumit Garg <sumit.garg@linaro.org> wrote:
> > >
> > > On Tue, 26 Nov 2024 at 14:03, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> > > >
> > > > On Mon, Nov 25, 2024 at 9:55 PM Amirreza Zarrabi
> > > > <quic_azarrabi@quicinc.com> wrote:
> > > > >
> > > > >
> > > > >
> > > > > On 11/25/2024 6:51 PM, Sumit Garg wrote:
> > > > > > On Mon, 25 Nov 2024 at 12:53, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> > > > > >>
> > > > > >> On Mon, Nov 25, 2024 at 7:14 AM Sumit Garg <sumit.garg@linaro.org> wrote:
> > > > > >>>
> > > > > >>> On Mon, 25 Nov 2024 at 03:00, Amirreza Zarrabi
> > > > > >>> <quic_azarrabi@quicinc.com> wrote:
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> Hi Sumit,
> > > > > >>>>
> > > > > >>>> Thank you so much for the comemnts :).
> > > > > >>>>
> > > > > >>>> On 11/23/2024 9:32 PM, Sumit Garg wrote:
> > > > > >>>>> Hi Amirreza,
> > > > > >>>>>
> > > > > >>>>> Thanks for proposing this.
> > > > > >>>>>
> > > > > >>>>> On Fri, 22 Nov 2024 at 06:38, Amirreza Zarrabi
> > > > > >>>>> <quic_azarrabi@quicinc.com> wrote:
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> On 11/21/2024 11:08 PM, Jens Wiklander wrote:
> > > > > >>>>>>
> > > > > >>>>>> Hi Jens,
> > > > > >>>>>>
> > > > > >>>>>>> Hi Amirreza,
> > > > > >>>>>>>
> > > > > >>>>>>> On Thu, Nov 21, 2024 at 2:37 AM Amirreza Zarrabi
> > > > > >>>>>>> <quic_azarrabi@quicinc.com> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>> The default context has a lifespan similar to the tee_device.
> > > > > >>>>>
> > > > > >>>>> Since it's associated with tee_device context, let's call it obvious
> > > > > >>>>> via renaming it as device context instead (s/def_ctx/dev_ctx/ in this
> > > > > >>>>> patch).
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>> Make sense, I'll rename it.
> > > > > >>>>
> > > > > >>>>>>>> It is used as a context for shared memory if the context to which the
> > > > > >>>>>>>> shared memory belongs is released, making the tee_shm an orphan.
> > > > > >>>>>>>> This allows the driver implementing shm_unregister to safely make
> > > > > >>>>>>>> subsequent calls, such as to a supplicant if needed.
> > > > > >>>>>>>>
> > > > > >>>>>>>> It also enables users to free the shared memory while the driver is
> > > > > >>>>>>>> blocked on unregister_tee_device safely.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Preferably, this should be used for all driver internal uses, using
> > > > > >>>>>>>> teedev_get_def_context rather than calling teedev_open.
> > > > > >>>>>
> > > > > >>>>> Makes sense to me.
> > > > > >>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>> Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
> > > > > >>>>>>>> ---
> > > > > >>>>>>>>  drivers/tee/optee/core.c    |  2 +-
> > > > > >>>>>>>>  drivers/tee/optee/ffa_abi.c |  2 +-
> > > > > >>>>>>>>  drivers/tee/optee/smc_abi.c |  2 +-
> > > > > >>>>>>>>  drivers/tee/tee_core.c      | 83 +++++++++++++++++++++++++++++----------------
> > > > > >>>>>>>>  drivers/tee/tee_private.h   |  3 --
> > > > > >>>>>>>>  drivers/tee/tee_shm.c       | 18 ++--------
> > > > > >>>>>>>>  include/linux/tee_core.h    | 15 ++++++++
> > > > > >>>>>>>>  include/linux/tee_drv.h     |  7 ----
> > > > > >>>>>>>>  8 files changed, 73 insertions(+), 59 deletions(-)
> > > > > >>>>>>>>
> > > > > >>>>>>>> diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
> > > > > >>>>>>>> index c75fddc83576..78d43d0c8014 100644
> > > > > >>>>>>>> --- a/drivers/tee/optee/core.c
> > > > > >>>>>>>> +++ b/drivers/tee/optee/core.c
> > > > > >>>>>>>> @@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
> > > > > >>>>>>>>
> > > > > >>>>>>>>         optee_notif_uninit(optee);
> > > > > >>>>>>>>         optee_shm_arg_cache_uninit(optee);
> > > > > >>>>>>>> -       teedev_close_context(optee->ctx);
> > > > > >>>>>>>> +
> > > > > >>>>>>>>         /*
> > > > > >>>>>>>>          * The two devices have to be unregistered before we can free the
> > > > > >>>>>>>>          * other resources.
> > > > > >>>>>>>> diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
> > > > > >>>>>>>> index f3af5666bb11..6ad94f0788ad 100644
> > > > > >>>>>>>> --- a/drivers/tee/optee/ffa_abi.c
> > > > > >>>>>>>> +++ b/drivers/tee/optee/ffa_abi.c
> > > > > >>>>>>>> @@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
> > > > > >>>>>>>>         optee_shm_arg_cache_init(optee, arg_cache_flags);
> > > > > >>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
> > > > > >>>>>>>>         ffa_dev_set_drvdata(ffa_dev, optee);
> > > > > >>>>>>>> -       ctx = teedev_open(optee->teedev);
> > > > > >>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
> > > > > >>>>>>>>         if (IS_ERR(ctx)) {
> > > > > >>>>>>>>                 rc = PTR_ERR(ctx);
> > > > > >>>>>>>>                 goto err_rhashtable_free;
> > > > > >>>>>>>> diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
> > > > > >>>>>>>> index e9456e3e74cc..c77a3e631d04 100644
> > > > > >>>>>>>> --- a/drivers/tee/optee/smc_abi.c
> > > > > >>>>>>>> +++ b/drivers/tee/optee/smc_abi.c
> > > > > >>>>>>>> @@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
> > > > > >>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
> > > > > >>>>>>>>
> > > > > >>>>>>>>         platform_set_drvdata(pdev, optee);
> > > > > >>>>>>>> -       ctx = teedev_open(optee->teedev);
> > > > > >>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
> > > > > >>>>>>>>         if (IS_ERR(ctx)) {
> > > > > >>>>>>>>                 rc = PTR_ERR(ctx);
> > > > > >>>>>>>>                 goto err_supp_uninit;
> > > > > >>>>>>>> diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
> > > > > >>>>>>>> index 93f3b330aec8..805e1336089d 100644
> > > > > >>>>>>>> --- a/drivers/tee/tee_core.c
> > > > > >>>>>>>> +++ b/drivers/tee/tee_core.c
> > > > > >>>>>>>> @@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> > > > > >>>>>>>>                 goto err;
> > > > > >>>>>>>>         }
> > > > > >>>>>>>>
> > > > > >>>>>>>> -       kref_init(&ctx->refcount);
> > > > > >>>>>>>>         ctx->teedev = teedev;
> > > > > >>>>>>>>         INIT_LIST_HEAD(&ctx->list_shm);
> > > > > >>>>>>>>         rc = teedev->desc->ops->open(ctx);
> > > > > >>>>>>>> @@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> > > > > >>>>>>>>  }
> > > > > >>>>>>>>  EXPORT_SYMBOL_GPL(teedev_open);
> > > > > >>>>>>>>
> > > > > >>>>>>>> -void teedev_ctx_get(struct tee_context *ctx)
> > > > > >>>>>>>> +struct tee_context *teedev_get_def_context(struct tee_device *teedev)
> > > > > >>>>>>>>  {
> > > > > >>>>>>>> -       if (ctx->releasing)
> > > > > >>>>>>>> -               return;
> > > > > >>>>>>>> +       int rc;
> > > > > >>>>>>>> +       struct tee_context *ctx = &teedev->def_ctx;
> > > > > >>>>>>>>
> > > > > >>>>>>>> -       kref_get(&ctx->refcount);
> > > > > >>>>>>>> -}
> > > > > >>>>>>>> +       ctx->teedev = teedev;
> > > > > >>>>>>>> +       INIT_LIST_HEAD(&ctx->list_shm);
> > > > > >>>>>>>> +       rc = teedev->desc->ops->open(ctx);
> > > > > >>>>>>>> +       if (rc)
> > > > > >>>>>>>> +               return ERR_PTR(rc);
> > > > > >>>>>>>
> > > > > >>>>>>> I think ctx->teedev and ctx->list_shm must always be initialized or
> > > > > >>>>>>> &teedev->def_ctx can't be used in teedev_close_context().
> > > > > >>>>>>
> > > > > >>>>>> True, but &teedev->def_ctx is never used in teedev_close_context().
> > > > > >>>>>> The closing of the &teedev->def_ctx simply ignored. So once opened,
> > > > > >>>>>> &teedev->def_ctx will always remain open until the tee_device is alive.
> > > > > >>>>>>
> > > > > >>>>>>> We could initialize teedev->def_ctx on the first call to teedev_open()
> > > > > >>>>>>> on that tee_device. We need a way to tell the
> > > > > >>>>>>> teedev->desc->ops->open() to the backed driver that it's initializing
> > > > > >>>>>>> the default context though, or optee_open() can't handle the
> > > > > >>>>>>> tee-supplicant case properly.
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>> That's a good point. This way, it is guaranteed that there is one def_ctx
> > > > > >>>>>> per teedev. There should be a way to tell the open() callback that it is
> > > > > >>>>>> a def_ctx, so it is not registered as a supplicant context.
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>>> Should we allow this function to be called more than once for each teedev?
> > > > > >>>>>>
> > > > > >>>>>> Yes, moving to teedev_open() will fix the issue.
> > > > > >>>>>>
> > > > > >>>>>>> Do we need serialization in this function if it's called after the
> > > > > >>>>>>> driver is probed?
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>> True. I'll make sure there is no race.
> > > > > >>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>> -static void teedev_ctx_release(struct kref *ref)
> > > > > >>>>>>>> -{
> > > > > >>>>>>>> -       struct tee_context *ctx = container_of(ref, struct tee_context,
> > > > > >>>>>>>> -                                              refcount);
> > > > > >>>>>>>> -       ctx->releasing = true;
> > > > > >>>>>>>> -       ctx->teedev->desc->ops->release(ctx);
> > > > > >>>>>>>> -       kfree(ctx);
> > > > > >>>>>>>> +       return ctx;
> > > > > >>>>>>>>  }
> > > > > >>>>>>>> +EXPORT_SYMBOL_GPL(teedev_get_def_context);
> > > > > >>>>>>>>
> > > > > >>>>>>>> -void teedev_ctx_put(struct tee_context *ctx)
> > > > > >>>>>>>> +void teedev_close_context(struct tee_context *ctx)
> > > > > >>>>>>>>  {
> > > > > >>>>>>>> -       if (ctx->releasing)
> > > > > >>>>>>>> +       struct tee_device *teedev = ctx->teedev;
> > > > > >>>>>>>> +       struct tee_shm *shm;
> > > > > >>>>>>>> +
> > > > > >>>>>>>> +       if (ctx == &teedev->def_ctx)
> > > > > >>>>>>>>                 return;
> > > > > >>>>>>>>
> > > > > >>>>>>>> -       kref_put(&ctx->refcount, teedev_ctx_release);
> > > > > >>>>>>>> -}
> > > > > >>>>>>>> +       teedev->desc->ops->release(ctx);
> > > > > >>>>>>>>
> > > > > >>>>>>>> -void teedev_close_context(struct tee_context *ctx)
> > > > > >>>>>>>> -{
> > > > > >>>>>>>> -       struct tee_device *teedev = ctx->teedev;
> > > > > >>>>>>>> +       mutex_lock(&teedev->mutex);
> > > > > >>>>>>>> +       list_for_each_entry(shm, &ctx->list_shm, link) {
> > > > > >>>>>>>> +               /* Context released. However, shm still holding a teedev reference.
> > > > > >>>>>>>> +                * Replace shm->ctx with the default context so that tee_shm_get_from_id()
> > > > > >>>>>>>> +                * fails (i.e. it is not accessible from userspace) but shm still
> > > > > >>>>>>>> +                * holds a valid context for further clean up, e.g. shm_unregister().
> > > > > >>>>>>>> +                */
> > > > > >>>>>>>
> > > > > >>>>>>> /*
> > > > > >>>>>>>  * Please format
> > > > > >>>>>>>  * multiline comments
> > > > > >>>>>>>  * like this. Please
> > > > > >>>>>>>  * keep the lines at
> > > > > >>>>>>>  * max 80 columns
> > > > > >>>>>>>  * here and at other
> > > > > >>>>>>>  * places in the patch-
> > > > > >>>>>>>  * set.
> > > > > >>>>>>>  */
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>> Ack.
> > > > > >>>>>>
> > > > > >>>>>>>> +               shm->ctx = &teedev->def_ctx;
> > > > > >>>>>>>
> > > > > >>>>>>> shm->ctx will always point to a valid context, even if it is the
> > > > > >>>>>>> default context. It seems that we can always get hold of the correct
> > > > > >>>>>>> teedev via shm->ctx->teedev. Do we need "tee: revert removal of
> > > > > >>>>>>> redundant teedev in struct tee_shm"?
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>> It was there in case we wanted to use NULL, but with def_ctx, it is not
> > > > > >>>>>> necessary. I am withdrawing that commit. :).
> > > > > >>>>>>
> > > > > >>>>>>> Shouldn't the shm be removed from the ctx->list_shm and be moved to
> > > > > >>>>>>> teedev->def_ctx.list_shm?
> > > > > >>>>>
> > > > > >>>>> +1
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>> Ack.
> > > > > >>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>> Not really. If we put shm in the teedev->def_ctx.list_shm, by the time
> > > > > >>>>>> we are closing the def_ctx, the list is guaranteed to be empty.
> > > > > >>>>>>
> > > > > >>>>>> However, I understand it is cleaner and more consistent to do that rather
> > > > > >>>>>> than making changes to tee_shm_put().
> > > > > >>>>>>
> > > > > >>>>>> I'll do it.
> > > > > >>>>>>
> > > > > >>>>>>>> +       }
> > > > > >>>>>>>> +       mutex_unlock(&teedev->mutex);
> > > > > >>>>>>>>
> > > > > >>>>>>>> -       teedev_ctx_put(ctx);
> > > > > >>>>>>>> +       kfree(ctx);
> > > > > >>>>>>>>         tee_device_put(teedev);
> > > > > >>>>>>>>  }
> > > > > >>>>>>>>  EXPORT_SYMBOL_GPL(teedev_close_context);
> > > > > >>>>>>>> @@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
> > > > > >>>>>>>>
> > > > > >>>>>>>>         teedev->desc = teedesc;
> > > > > >>>>>>>>         teedev->pool = pool;
> > > > > >>>>>>>> +       /* Only open default context when teedev_get_def_context() called. */
> > > > > >>>>>>>> +       teedev->def_ctx.teedev = NULL;
> > > > > >>>>>
> > > > > >>>>> Why don't you open the device context here only? This will associate
> > > > > >>>>> it automatically with teedev lifespan and then
> > > > > >>>>> teedev_get_def_context() will just return a reference to that.
> > > > > >>>>>
> > > > > >>>>> -Sumit
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>> So my assumption is that the tee_devic_alloc() is called as part of
> > > > > >>>> the driver initialization; there is no guarantee that at this time the
> > > > > >>>> driver is actually ready to accept any open() callback.
> > > > > >>>>
> > > > > >>>
> > > > > >>> The drivers should be able to handle open() callback since we already
> > > > > >>> check for !teedesc->ops->open in the beginning of tee_devic_alloc().
> > > > > >>> Also, we need to open a device context for !TEE_DESC_PRIVILEGED such
> > > > > >>> that we don't open a supplicant device context there.
> > > > > >>
> > > > > >> It would be nice to have the device context fully initialized when the
> > > > > >> probe function returns. How about adding a "bool is_dev_ctx" to struct
> > > > > >> tee_context so the open() callback can tell that this is a special
> > > > > >> tee_contex?
> > > > > >
> > > > > > Sure, that will be useful to distinguish the device context from
> > > > > > normal client context.
> > > > > >
> > > > > > -Sumit
> > > > > >
> > > > >
> > > > > So, as far as the open() callback, I do not believe checking if it is not null
> > > > > is reasonable for calling it here. Most drivers allocate resources and then
> > > > > initialize them. So, assume these steps for a TEE driver:
> > > > >  (1) allocate internal data structures,
> > > > >  (2) allocate the device,
> > > > >  (3) initialize the internal data structurse and then
> > > > >  (4) register the device.
> > > > >
> > > > > Having these steps for a backend driver means that if you call open() at
> > > > > step (2), the internal data structures are not ready.
> > >
> > > As part of tee_device_alloc(), every driver has to pass "const struct
> > > tee_desc *teedesc" fully initialized. Which internal data structures
> > > are you referring too? Is there any upstream example?
> >
> > It's reasonable to wait with the open() callback until step 4 above,
> > which should correspond with the tee_device_register() call. Data
> > written only once doesn't need serialized access if the fields are
> > only accessed after they have been fully initialized.
>
> Fair enough, I can live with the device context opened after registering it.
>
> >
> > >
> > > > >
> > > > > I was originally thinking of going with Jens' suggestion to open dev_ctx in
> > > > > the teedev_open(), and use a flag to distinguish the type of context for
> > > > > the open() callback
> > > > >
> > > > > What about this:
> > > > > Open the dev_ctx in the tee_device_register(), at the last step before
> > > > > setting the TEE_DEVICE_FLAG_REGISTERED flag. Then the open() callback can
> > > > > check for this flag to determine if it is a normal context or dev_ctx.
> > > > > If the open() is called while the device has not been registered, it should
> > > > > handle it differently
> > > >
> > > > That makes sense, the driver should be prepared to handle open() calls
> > > > after tee_device_register() anyway.
> > > > However, there is no serialization of the flags field in struct
> > > > tee_device. Hmm, would it be too hacky for the open() callback to
> > > > check if &ctx->teedev.dev_ctx == ctx? We could add a helper function
> > > > to wrap that check.
> > > >
> > >
> > > Your suggested change requires every driver to update open() callback
> > > and later other callbacks may have to support it too. IMHO, only
> > > teedev_get_dev_ctx() should be able to return a reference to device
> > > context for usage within the TEE and the implementation driver.
> >
> > Yes, but it's only the OP-TEE driver that needs anything special. It
> > looks like the others can be left unchanged.
>
> I suppose it's most likely the upcoming QTEE driver requiring it.
>
> >
> > >
> > > I am still not able to understand why the following won't work with a
> > > clear lifetime for the device context?
> > >
> > > tee_device_alloc()
> > >   -> if (!(teedesc->flags & TEE_DESC_PRIVILEGED))
> > > desc->ops->open(&teedev->dev_ctx);
> >
> > We must also have a fully initialized dev_ctx for the supplicant
> > device.
>
> Currently I only see following for OP-TEE driver:
>
> ctx = teedev_open(optee->teedev);
>
> And I can't see anything like below:
>
> ctx = teedev_open(optee->supp_teedev);
>
> Where do you think that the dev_ctx is required for a supplicant
> device? AFAICS, currently opening a context with the supplicant device
> means that the supplicant daemon is available to handle RPCs which
> won't be possible during OP-TEE driver probe. Am I missing something?

One reason for initializing dev_ctx for all tee_devices is in
teedev_close_context(), where the tee_shms still active are
transferred to dev_ctx. The teedev member was re-introduced in this
patch set, but it can be removed again if we can depend on the dev_ctx
to always be available in teedev_close_context(). Even the
tee-supplicant may close its tee_context with active tee_shms at some
point. It might be possible to use half-baked dev_ctx, but then we'd
be burdened with keeping track of which dev_ctx can be used for what.
We want as few special cases as possible.

>
> > I'd rather delay the open() callback until
> > tee_device_register() since the dev_ctx is guaranteed not to be needed
> > before that.
>
> Okay, the updated call chain can look like:
>
> tee_device_register()
>   -> if (!(teedev->desc->flags & TEE_DESC_PRIVILEGED))
>          desc->ops->open(&teedev->dev_ctx);
> >
> > >
> > > tee_device_put()
> > >   -> if (teedev->dev_ctx) desc->ops->release(&teedev->dev_ctx);
> >
> > teedev->dev_ctx is supposed to be embedded in struct tee_device, so
> > the if isn't needed.
>
> I added "if" to cover the case when dev_ctx is not initialized for the
> supplicant device.

OK.

Cheers,
Jens

>
> -Sumit
>
> >
> > Cheers,
> > Jens
> >
> > >
> > > -Sumit
> > >
> > > > Cheers,
> > > > Jens
> > > >
> > > > >
> > > > > - Amir
> > > > >
> > > > > >>
> > > > > >> Cheers,
> > > > > >> Jens
> > > > > >>
> > > > > >>>
> > > > > >>> -Sumit

Re: [PATCH RFC 3/3] tee: introduce orphan tee_shm and default context

[Amirreza Zarrabi]

On 11/27/2024 6:22 PM, Jens Wiklander wrote:
> On Wed, Nov 27, 2024 at 7:02 AM Sumit Garg <sumit.garg@linaro.org> wrote:
>>
>> On Tue, 26 Nov 2024 at 20:52, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>>
>>> On Tue, Nov 26, 2024 at 1:27 PM Sumit Garg <sumit.garg@linaro.org> wrote:
>>>>
>>>> On Tue, 26 Nov 2024 at 14:03, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>>>>
>>>>> On Mon, Nov 25, 2024 at 9:55 PM Amirreza Zarrabi
>>>>> <quic_azarrabi@quicinc.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 11/25/2024 6:51 PM, Sumit Garg wrote:
>>>>>>> On Mon, 25 Nov 2024 at 12:53, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>>>>>>>
>>>>>>>> On Mon, Nov 25, 2024 at 7:14 AM Sumit Garg <sumit.garg@linaro.org> wrote:
>>>>>>>>>
>>>>>>>>> On Mon, 25 Nov 2024 at 03:00, Amirreza Zarrabi
>>>>>>>>> <quic_azarrabi@quicinc.com> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Sumit,
>>>>>>>>>>
>>>>>>>>>> Thank you so much for the comemnts :).
>>>>>>>>>>
>>>>>>>>>> On 11/23/2024 9:32 PM, Sumit Garg wrote:
>>>>>>>>>>> Hi Amirreza,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for proposing this.
>>>>>>>>>>>
>>>>>>>>>>> On Fri, 22 Nov 2024 at 06:38, Amirreza Zarrabi
>>>>>>>>>>> <quic_azarrabi@quicinc.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 11/21/2024 11:08 PM, Jens Wiklander wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Jens,
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Amirreza,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Nov 21, 2024 at 2:37 AM Amirreza Zarrabi
>>>>>>>>>>>>> <quic_azarrabi@quicinc.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The default context has a lifespan similar to the tee_device.
>>>>>>>>>>>
>>>>>>>>>>> Since it's associated with tee_device context, let's call it obvious
>>>>>>>>>>> via renaming it as device context instead (s/def_ctx/dev_ctx/ in this
>>>>>>>>>>> patch).
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Make sense, I'll rename it.
>>>>>>>>>>
>>>>>>>>>>>>>> It is used as a context for shared memory if the context to which the
>>>>>>>>>>>>>> shared memory belongs is released, making the tee_shm an orphan.
>>>>>>>>>>>>>> This allows the driver implementing shm_unregister to safely make
>>>>>>>>>>>>>> subsequent calls, such as to a supplicant if needed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It also enables users to free the shared memory while the driver is
>>>>>>>>>>>>>> blocked on unregister_tee_device safely.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Preferably, this should be used for all driver internal uses, using
>>>>>>>>>>>>>> teedev_get_def_context rather than calling teedev_open.
>>>>>>>>>>>
>>>>>>>>>>> Makes sense to me.
>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>  drivers/tee/optee/core.c    |  2 +-
>>>>>>>>>>>>>>  drivers/tee/optee/ffa_abi.c |  2 +-
>>>>>>>>>>>>>>  drivers/tee/optee/smc_abi.c |  2 +-
>>>>>>>>>>>>>>  drivers/tee/tee_core.c      | 83 +++++++++++++++++++++++++++++----------------
>>>>>>>>>>>>>>  drivers/tee/tee_private.h   |  3 --
>>>>>>>>>>>>>>  drivers/tee/tee_shm.c       | 18 ++--------
>>>>>>>>>>>>>>  include/linux/tee_core.h    | 15 ++++++++
>>>>>>>>>>>>>>  include/linux/tee_drv.h     |  7 ----
>>>>>>>>>>>>>>  8 files changed, 73 insertions(+), 59 deletions(-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
>>>>>>>>>>>>>> index c75fddc83576..78d43d0c8014 100644
>>>>>>>>>>>>>> --- a/drivers/tee/optee/core.c
>>>>>>>>>>>>>> +++ b/drivers/tee/optee/core.c
>>>>>>>>>>>>>> @@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>         optee_notif_uninit(optee);
>>>>>>>>>>>>>>         optee_shm_arg_cache_uninit(optee);
>>>>>>>>>>>>>> -       teedev_close_context(optee->ctx);
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>         /*
>>>>>>>>>>>>>>          * The two devices have to be unregistered before we can free the
>>>>>>>>>>>>>>          * other resources.
>>>>>>>>>>>>>> diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
>>>>>>>>>>>>>> index f3af5666bb11..6ad94f0788ad 100644
>>>>>>>>>>>>>> --- a/drivers/tee/optee/ffa_abi.c
>>>>>>>>>>>>>> +++ b/drivers/tee/optee/ffa_abi.c
>>>>>>>>>>>>>> @@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
>>>>>>>>>>>>>>         optee_shm_arg_cache_init(optee, arg_cache_flags);
>>>>>>>>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
>>>>>>>>>>>>>>         ffa_dev_set_drvdata(ffa_dev, optee);
>>>>>>>>>>>>>> -       ctx = teedev_open(optee->teedev);
>>>>>>>>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
>>>>>>>>>>>>>>         if (IS_ERR(ctx)) {
>>>>>>>>>>>>>>                 rc = PTR_ERR(ctx);
>>>>>>>>>>>>>>                 goto err_rhashtable_free;
>>>>>>>>>>>>>> diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
>>>>>>>>>>>>>> index e9456e3e74cc..c77a3e631d04 100644
>>>>>>>>>>>>>> --- a/drivers/tee/optee/smc_abi.c
>>>>>>>>>>>>>> +++ b/drivers/tee/optee/smc_abi.c
>>>>>>>>>>>>>> @@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
>>>>>>>>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>         platform_set_drvdata(pdev, optee);
>>>>>>>>>>>>>> -       ctx = teedev_open(optee->teedev);
>>>>>>>>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
>>>>>>>>>>>>>>         if (IS_ERR(ctx)) {
>>>>>>>>>>>>>>                 rc = PTR_ERR(ctx);
>>>>>>>>>>>>>>                 goto err_supp_uninit;
>>>>>>>>>>>>>> diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
>>>>>>>>>>>>>> index 93f3b330aec8..805e1336089d 100644
>>>>>>>>>>>>>> --- a/drivers/tee/tee_core.c
>>>>>>>>>>>>>> +++ b/drivers/tee/tee_core.c
>>>>>>>>>>>>>> @@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
>>>>>>>>>>>>>>                 goto err;
>>>>>>>>>>>>>>         }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -       kref_init(&ctx->refcount);
>>>>>>>>>>>>>>         ctx->teedev = teedev;
>>>>>>>>>>>>>>         INIT_LIST_HEAD(&ctx->list_shm);
>>>>>>>>>>>>>>         rc = teedev->desc->ops->open(ctx);
>>>>>>>>>>>>>> @@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
>>>>>>>>>>>>>>  }
>>>>>>>>>>>>>>  EXPORT_SYMBOL_GPL(teedev_open);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -void teedev_ctx_get(struct tee_context *ctx)
>>>>>>>>>>>>>> +struct tee_context *teedev_get_def_context(struct tee_device *teedev)
>>>>>>>>>>>>>>  {
>>>>>>>>>>>>>> -       if (ctx->releasing)
>>>>>>>>>>>>>> -               return;
>>>>>>>>>>>>>> +       int rc;
>>>>>>>>>>>>>> +       struct tee_context *ctx = &teedev->def_ctx;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -       kref_get(&ctx->refcount);
>>>>>>>>>>>>>> -}
>>>>>>>>>>>>>> +       ctx->teedev = teedev;
>>>>>>>>>>>>>> +       INIT_LIST_HEAD(&ctx->list_shm);
>>>>>>>>>>>>>> +       rc = teedev->desc->ops->open(ctx);
>>>>>>>>>>>>>> +       if (rc)
>>>>>>>>>>>>>> +               return ERR_PTR(rc);
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think ctx->teedev and ctx->list_shm must always be initialized or
>>>>>>>>>>>>> &teedev->def_ctx can't be used in teedev_close_context().
>>>>>>>>>>>>
>>>>>>>>>>>> True, but &teedev->def_ctx is never used in teedev_close_context().
>>>>>>>>>>>> The closing of the &teedev->def_ctx simply ignored. So once opened,
>>>>>>>>>>>> &teedev->def_ctx will always remain open until the tee_device is alive.
>>>>>>>>>>>>
>>>>>>>>>>>>> We could initialize teedev->def_ctx on the first call to teedev_open()
>>>>>>>>>>>>> on that tee_device. We need a way to tell the
>>>>>>>>>>>>> teedev->desc->ops->open() to the backed driver that it's initializing
>>>>>>>>>>>>> the default context though, or optee_open() can't handle the
>>>>>>>>>>>>> tee-supplicant case properly.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> That's a good point. This way, it is guaranteed that there is one def_ctx
>>>>>>>>>>>> per teedev. There should be a way to tell the open() callback that it is
>>>>>>>>>>>> a def_ctx, so it is not registered as a supplicant context.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Should we allow this function to be called more than once for each teedev?
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, moving to teedev_open() will fix the issue.
>>>>>>>>>>>>
>>>>>>>>>>>>> Do we need serialization in this function if it's called after the
>>>>>>>>>>>>> driver is probed?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> True. I'll make sure there is no race.
>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -static void teedev_ctx_release(struct kref *ref)
>>>>>>>>>>>>>> -{
>>>>>>>>>>>>>> -       struct tee_context *ctx = container_of(ref, struct tee_context,
>>>>>>>>>>>>>> -                                              refcount);
>>>>>>>>>>>>>> -       ctx->releasing = true;
>>>>>>>>>>>>>> -       ctx->teedev->desc->ops->release(ctx);
>>>>>>>>>>>>>> -       kfree(ctx);
>>>>>>>>>>>>>> +       return ctx;
>>>>>>>>>>>>>>  }
>>>>>>>>>>>>>> +EXPORT_SYMBOL_GPL(teedev_get_def_context);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -void teedev_ctx_put(struct tee_context *ctx)
>>>>>>>>>>>>>> +void teedev_close_context(struct tee_context *ctx)
>>>>>>>>>>>>>>  {
>>>>>>>>>>>>>> -       if (ctx->releasing)
>>>>>>>>>>>>>> +       struct tee_device *teedev = ctx->teedev;
>>>>>>>>>>>>>> +       struct tee_shm *shm;
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>> +       if (ctx == &teedev->def_ctx)
>>>>>>>>>>>>>>                 return;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -       kref_put(&ctx->refcount, teedev_ctx_release);
>>>>>>>>>>>>>> -}
>>>>>>>>>>>>>> +       teedev->desc->ops->release(ctx);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -void teedev_close_context(struct tee_context *ctx)
>>>>>>>>>>>>>> -{
>>>>>>>>>>>>>> -       struct tee_device *teedev = ctx->teedev;
>>>>>>>>>>>>>> +       mutex_lock(&teedev->mutex);
>>>>>>>>>>>>>> +       list_for_each_entry(shm, &ctx->list_shm, link) {
>>>>>>>>>>>>>> +               /* Context released. However, shm still holding a teedev reference.
>>>>>>>>>>>>>> +                * Replace shm->ctx with the default context so that tee_shm_get_from_id()
>>>>>>>>>>>>>> +                * fails (i.e. it is not accessible from userspace) but shm still
>>>>>>>>>>>>>> +                * holds a valid context for further clean up, e.g. shm_unregister().
>>>>>>>>>>>>>> +                */
>>>>>>>>>>>>>
>>>>>>>>>>>>> /*
>>>>>>>>>>>>>  * Please format
>>>>>>>>>>>>>  * multiline comments
>>>>>>>>>>>>>  * like this. Please
>>>>>>>>>>>>>  * keep the lines at
>>>>>>>>>>>>>  * max 80 columns
>>>>>>>>>>>>>  * here and at other
>>>>>>>>>>>>>  * places in the patch-
>>>>>>>>>>>>>  * set.
>>>>>>>>>>>>>  */
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Ack.
>>>>>>>>>>>>
>>>>>>>>>>>>>> +               shm->ctx = &teedev->def_ctx;
>>>>>>>>>>>>>
>>>>>>>>>>>>> shm->ctx will always point to a valid context, even if it is the
>>>>>>>>>>>>> default context. It seems that we can always get hold of the correct
>>>>>>>>>>>>> teedev via shm->ctx->teedev. Do we need "tee: revert removal of
>>>>>>>>>>>>> redundant teedev in struct tee_shm"?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> It was there in case we wanted to use NULL, but with def_ctx, it is not
>>>>>>>>>>>> necessary. I am withdrawing that commit. :).
>>>>>>>>>>>>
>>>>>>>>>>>>> Shouldn't the shm be removed from the ctx->list_shm and be moved to
>>>>>>>>>>>>> teedev->def_ctx.list_shm?
>>>>>>>>>>>
>>>>>>>>>>> +1
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Ack.
>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Not really. If we put shm in the teedev->def_ctx.list_shm, by the time
>>>>>>>>>>>> we are closing the def_ctx, the list is guaranteed to be empty.
>>>>>>>>>>>>
>>>>>>>>>>>> However, I understand it is cleaner and more consistent to do that rather
>>>>>>>>>>>> than making changes to tee_shm_put().
>>>>>>>>>>>>
>>>>>>>>>>>> I'll do it.
>>>>>>>>>>>>
>>>>>>>>>>>>>> +       }
>>>>>>>>>>>>>> +       mutex_unlock(&teedev->mutex);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -       teedev_ctx_put(ctx);
>>>>>>>>>>>>>> +       kfree(ctx);
>>>>>>>>>>>>>>         tee_device_put(teedev);
>>>>>>>>>>>>>>  }
>>>>>>>>>>>>>>  EXPORT_SYMBOL_GPL(teedev_close_context);
>>>>>>>>>>>>>> @@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>         teedev->desc = teedesc;
>>>>>>>>>>>>>>         teedev->pool = pool;
>>>>>>>>>>>>>> +       /* Only open default context when teedev_get_def_context() called. */
>>>>>>>>>>>>>> +       teedev->def_ctx.teedev = NULL;
>>>>>>>>>>>
>>>>>>>>>>> Why don't you open the device context here only? This will associate
>>>>>>>>>>> it automatically with teedev lifespan and then
>>>>>>>>>>> teedev_get_def_context() will just return a reference to that.
>>>>>>>>>>>
>>>>>>>>>>> -Sumit
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> So my assumption is that the tee_devic_alloc() is called as part of
>>>>>>>>>> the driver initialization; there is no guarantee that at this time the
>>>>>>>>>> driver is actually ready to accept any open() callback.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The drivers should be able to handle open() callback since we already
>>>>>>>>> check for !teedesc->ops->open in the beginning of tee_devic_alloc().
>>>>>>>>> Also, we need to open a device context for !TEE_DESC_PRIVILEGED such
>>>>>>>>> that we don't open a supplicant device context there.
>>>>>>>>
>>>>>>>> It would be nice to have the device context fully initialized when the
>>>>>>>> probe function returns. How about adding a "bool is_dev_ctx" to struct
>>>>>>>> tee_context so the open() callback can tell that this is a special
>>>>>>>> tee_contex?
>>>>>>>
>>>>>>> Sure, that will be useful to distinguish the device context from
>>>>>>> normal client context.
>>>>>>>
>>>>>>> -Sumit
>>>>>>>
>>>>>>
>>>>>> So, as far as the open() callback, I do not believe checking if it is not null
>>>>>> is reasonable for calling it here. Most drivers allocate resources and then
>>>>>> initialize them. So, assume these steps for a TEE driver:
>>>>>>  (1) allocate internal data structures,
>>>>>>  (2) allocate the device,
>>>>>>  (3) initialize the internal data structurse and then
>>>>>>  (4) register the device.
>>>>>>
>>>>>> Having these steps for a backend driver means that if you call open() at
>>>>>> step (2), the internal data structures are not ready.
>>>>
>>>> As part of tee_device_alloc(), every driver has to pass "const struct
>>>> tee_desc *teedesc" fully initialized. Which internal data structures
>>>> are you referring too? Is there any upstream example?
>>>
>>> It's reasonable to wait with the open() callback until step 4 above,
>>> which should correspond with the tee_device_register() call. Data
>>> written only once doesn't need serialized access if the fields are
>>> only accessed after they have been fully initialized.
>>
>> Fair enough, I can live with the device context opened after registering it.
>>
>>>
>>>>
>>>>>>
>>>>>> I was originally thinking of going with Jens' suggestion to open dev_ctx in
>>>>>> the teedev_open(), and use a flag to distinguish the type of context for
>>>>>> the open() callback
>>>>>>
>>>>>> What about this:
>>>>>> Open the dev_ctx in the tee_device_register(), at the last step before
>>>>>> setting the TEE_DEVICE_FLAG_REGISTERED flag. Then the open() callback can
>>>>>> check for this flag to determine if it is a normal context or dev_ctx.
>>>>>> If the open() is called while the device has not been registered, it should
>>>>>> handle it differently
>>>>>
>>>>> That makes sense, the driver should be prepared to handle open() calls
>>>>> after tee_device_register() anyway.
>>>>> However, there is no serialization of the flags field in struct
>>>>> tee_device. Hmm, would it be too hacky for the open() callback to
>>>>> check if &ctx->teedev.dev_ctx == ctx? We could add a helper function
>>>>> to wrap that check.
>>>>>
>>>>
>>>> Your suggested change requires every driver to update open() callback
>>>> and later other callbacks may have to support it too. IMHO, only
>>>> teedev_get_dev_ctx() should be able to return a reference to device
>>>> context for usage within the TEE and the implementation driver.
>>>
>>> Yes, but it's only the OP-TEE driver that needs anything special. It
>>> looks like the others can be left unchanged.
>>
>> I suppose it's most likely the upcoming QTEE driver requiring it.
>>
>>>
>>>>
>>>> I am still not able to understand why the following won't work with a
>>>> clear lifetime for the device context?
>>>>
>>>> tee_device_alloc()
>>>>   -> if (!(teedesc->flags & TEE_DESC_PRIVILEGED))
>>>> desc->ops->open(&teedev->dev_ctx);
>>>
>>> We must also have a fully initialized dev_ctx for the supplicant
>>> device.
>>
>> Currently I only see following for OP-TEE driver:
>>
>> ctx = teedev_open(optee->teedev);
>>
>> And I can't see anything like below:
>>
>> ctx = teedev_open(optee->supp_teedev);
>>
>> Where do you think that the dev_ctx is required for a supplicant
>> device? AFAICS, currently opening a context with the supplicant device
>> means that the supplicant daemon is available to handle RPCs which
>> won't be possible during OP-TEE driver probe. Am I missing something?
> 
> One reason for initializing dev_ctx for all tee_devices is in
> teedev_close_context(), where the tee_shms still active are
> transferred to dev_ctx. The teedev member was re-introduced in this
> patch set, but it can be removed again if we can depend on the dev_ctx
> to always be available in teedev_close_context(). Even the
> tee-supplicant may close its tee_context with active tee_shms at some
> point. It might be possible to use half-baked dev_ctx, but then we'd
> be burdened with keeping track of which dev_ctx can be used for what.
> We want as few special cases as possible.
> 

Additionally, Jens suggested checking something like (&ctx->teedev.dev_ctx == ctx)
in the open() callback to ensure that dev_ctx is not accidentally registered as
a supplicant context. This helps avoid the issue you're referring to.

Regards,
Amir

>>
>>> I'd rather delay the open() callback until
>>> tee_device_register() since the dev_ctx is guaranteed not to be needed
>>> before that.
>>
>> Okay, the updated call chain can look like:
>>
>> tee_device_register()
>>   -> if (!(teedev->desc->flags & TEE_DESC_PRIVILEGED))
>>          desc->ops->open(&teedev->dev_ctx);
>>>
>>>>
>>>> tee_device_put()
>>>>   -> if (teedev->dev_ctx) desc->ops->release(&teedev->dev_ctx);
>>>
>>> teedev->dev_ctx is supposed to be embedded in struct tee_device, so
>>> the if isn't needed.
>>
>> I added "if" to cover the case when dev_ctx is not initialized for the
>> supplicant device.
> 
> OK.
> 
> Cheers,
> Jens
> 
>>
>> -Sumit
>>
>>>
>>> Cheers,
>>> Jens
>>>
>>>>
>>>> -Sumit
>>>>
>>>>> Cheers,
>>>>> Jens
>>>>>
>>>>>>
>>>>>> - Amir
>>>>>>
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Jens
>>>>>>>>
>>>>>>>>>
>>>>>>>>> -Sumit

Re: [PATCH RFC 3/3] tee: introduce orphan tee_shm and default context

[Sumit Garg]

On Wed, 27 Nov 2024 at 12:52, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>
> On Wed, Nov 27, 2024 at 7:02 AM Sumit Garg <sumit.garg@linaro.org> wrote:
> >
> > On Tue, 26 Nov 2024 at 20:52, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> > >
> > > On Tue, Nov 26, 2024 at 1:27 PM Sumit Garg <sumit.garg@linaro.org> wrote:
> > > >
> > > > On Tue, 26 Nov 2024 at 14:03, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> > > > >
> > > > > On Mon, Nov 25, 2024 at 9:55 PM Amirreza Zarrabi
> > > > > <quic_azarrabi@quicinc.com> wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 11/25/2024 6:51 PM, Sumit Garg wrote:
> > > > > > > On Mon, 25 Nov 2024 at 12:53, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> > > > > > >>
> > > > > > >> On Mon, Nov 25, 2024 at 7:14 AM Sumit Garg <sumit.garg@linaro.org> wrote:
> > > > > > >>>
> > > > > > >>> On Mon, 25 Nov 2024 at 03:00, Amirreza Zarrabi
> > > > > > >>> <quic_azarrabi@quicinc.com> wrote:
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> Hi Sumit,
> > > > > > >>>>
> > > > > > >>>> Thank you so much for the comemnts :).
> > > > > > >>>>
> > > > > > >>>> On 11/23/2024 9:32 PM, Sumit Garg wrote:
> > > > > > >>>>> Hi Amirreza,
> > > > > > >>>>>
> > > > > > >>>>> Thanks for proposing this.
> > > > > > >>>>>
> > > > > > >>>>> On Fri, 22 Nov 2024 at 06:38, Amirreza Zarrabi
> > > > > > >>>>> <quic_azarrabi@quicinc.com> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> On 11/21/2024 11:08 PM, Jens Wiklander wrote:
> > > > > > >>>>>>
> > > > > > >>>>>> Hi Jens,
> > > > > > >>>>>>
> > > > > > >>>>>>> Hi Amirreza,
> > > > > > >>>>>>>
> > > > > > >>>>>>> On Thu, Nov 21, 2024 at 2:37 AM Amirreza Zarrabi
> > > > > > >>>>>>> <quic_azarrabi@quicinc.com> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> The default context has a lifespan similar to the tee_device.
> > > > > > >>>>>
> > > > > > >>>>> Since it's associated with tee_device context, let's call it obvious
> > > > > > >>>>> via renaming it as device context instead (s/def_ctx/dev_ctx/ in this
> > > > > > >>>>> patch).
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>> Make sense, I'll rename it.
> > > > > > >>>>
> > > > > > >>>>>>>> It is used as a context for shared memory if the context to which the
> > > > > > >>>>>>>> shared memory belongs is released, making the tee_shm an orphan.
> > > > > > >>>>>>>> This allows the driver implementing shm_unregister to safely make
> > > > > > >>>>>>>> subsequent calls, such as to a supplicant if needed.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> It also enables users to free the shared memory while the driver is
> > > > > > >>>>>>>> blocked on unregister_tee_device safely.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Preferably, this should be used for all driver internal uses, using
> > > > > > >>>>>>>> teedev_get_def_context rather than calling teedev_open.
> > > > > > >>>>>
> > > > > > >>>>> Makes sense to me.
> > > > > > >>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
> > > > > > >>>>>>>> ---
> > > > > > >>>>>>>>  drivers/tee/optee/core.c    |  2 +-
> > > > > > >>>>>>>>  drivers/tee/optee/ffa_abi.c |  2 +-
> > > > > > >>>>>>>>  drivers/tee/optee/smc_abi.c |  2 +-
> > > > > > >>>>>>>>  drivers/tee/tee_core.c      | 83 +++++++++++++++++++++++++++++----------------
> > > > > > >>>>>>>>  drivers/tee/tee_private.h   |  3 --
> > > > > > >>>>>>>>  drivers/tee/tee_shm.c       | 18 ++--------
> > > > > > >>>>>>>>  include/linux/tee_core.h    | 15 ++++++++
> > > > > > >>>>>>>>  include/linux/tee_drv.h     |  7 ----
> > > > > > >>>>>>>>  8 files changed, 73 insertions(+), 59 deletions(-)
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
> > > > > > >>>>>>>> index c75fddc83576..78d43d0c8014 100644
> > > > > > >>>>>>>> --- a/drivers/tee/optee/core.c
> > > > > > >>>>>>>> +++ b/drivers/tee/optee/core.c
> > > > > > >>>>>>>> @@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>         optee_notif_uninit(optee);
> > > > > > >>>>>>>>         optee_shm_arg_cache_uninit(optee);
> > > > > > >>>>>>>> -       teedev_close_context(optee->ctx);
> > > > > > >>>>>>>> +
> > > > > > >>>>>>>>         /*
> > > > > > >>>>>>>>          * The two devices have to be unregistered before we can free the
> > > > > > >>>>>>>>          * other resources.
> > > > > > >>>>>>>> diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
> > > > > > >>>>>>>> index f3af5666bb11..6ad94f0788ad 100644
> > > > > > >>>>>>>> --- a/drivers/tee/optee/ffa_abi.c
> > > > > > >>>>>>>> +++ b/drivers/tee/optee/ffa_abi.c
> > > > > > >>>>>>>> @@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
> > > > > > >>>>>>>>         optee_shm_arg_cache_init(optee, arg_cache_flags);
> > > > > > >>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
> > > > > > >>>>>>>>         ffa_dev_set_drvdata(ffa_dev, optee);
> > > > > > >>>>>>>> -       ctx = teedev_open(optee->teedev);
> > > > > > >>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
> > > > > > >>>>>>>>         if (IS_ERR(ctx)) {
> > > > > > >>>>>>>>                 rc = PTR_ERR(ctx);
> > > > > > >>>>>>>>                 goto err_rhashtable_free;
> > > > > > >>>>>>>> diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
> > > > > > >>>>>>>> index e9456e3e74cc..c77a3e631d04 100644
> > > > > > >>>>>>>> --- a/drivers/tee/optee/smc_abi.c
> > > > > > >>>>>>>> +++ b/drivers/tee/optee/smc_abi.c
> > > > > > >>>>>>>> @@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
> > > > > > >>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>         platform_set_drvdata(pdev, optee);
> > > > > > >>>>>>>> -       ctx = teedev_open(optee->teedev);
> > > > > > >>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
> > > > > > >>>>>>>>         if (IS_ERR(ctx)) {
> > > > > > >>>>>>>>                 rc = PTR_ERR(ctx);
> > > > > > >>>>>>>>                 goto err_supp_uninit;
> > > > > > >>>>>>>> diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
> > > > > > >>>>>>>> index 93f3b330aec8..805e1336089d 100644
> > > > > > >>>>>>>> --- a/drivers/tee/tee_core.c
> > > > > > >>>>>>>> +++ b/drivers/tee/tee_core.c
> > > > > > >>>>>>>> @@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> > > > > > >>>>>>>>                 goto err;
> > > > > > >>>>>>>>         }
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> -       kref_init(&ctx->refcount);
> > > > > > >>>>>>>>         ctx->teedev = teedev;
> > > > > > >>>>>>>>         INIT_LIST_HEAD(&ctx->list_shm);
> > > > > > >>>>>>>>         rc = teedev->desc->ops->open(ctx);
> > > > > > >>>>>>>> @@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> > > > > > >>>>>>>>  }
> > > > > > >>>>>>>>  EXPORT_SYMBOL_GPL(teedev_open);
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> -void teedev_ctx_get(struct tee_context *ctx)
> > > > > > >>>>>>>> +struct tee_context *teedev_get_def_context(struct tee_device *teedev)
> > > > > > >>>>>>>>  {
> > > > > > >>>>>>>> -       if (ctx->releasing)
> > > > > > >>>>>>>> -               return;
> > > > > > >>>>>>>> +       int rc;
> > > > > > >>>>>>>> +       struct tee_context *ctx = &teedev->def_ctx;
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> -       kref_get(&ctx->refcount);
> > > > > > >>>>>>>> -}
> > > > > > >>>>>>>> +       ctx->teedev = teedev;
> > > > > > >>>>>>>> +       INIT_LIST_HEAD(&ctx->list_shm);
> > > > > > >>>>>>>> +       rc = teedev->desc->ops->open(ctx);
> > > > > > >>>>>>>> +       if (rc)
> > > > > > >>>>>>>> +               return ERR_PTR(rc);
> > > > > > >>>>>>>
> > > > > > >>>>>>> I think ctx->teedev and ctx->list_shm must always be initialized or
> > > > > > >>>>>>> &teedev->def_ctx can't be used in teedev_close_context().
> > > > > > >>>>>>
> > > > > > >>>>>> True, but &teedev->def_ctx is never used in teedev_close_context().
> > > > > > >>>>>> The closing of the &teedev->def_ctx simply ignored. So once opened,
> > > > > > >>>>>> &teedev->def_ctx will always remain open until the tee_device is alive.
> > > > > > >>>>>>
> > > > > > >>>>>>> We could initialize teedev->def_ctx on the first call to teedev_open()
> > > > > > >>>>>>> on that tee_device. We need a way to tell the
> > > > > > >>>>>>> teedev->desc->ops->open() to the backed driver that it's initializing
> > > > > > >>>>>>> the default context though, or optee_open() can't handle the
> > > > > > >>>>>>> tee-supplicant case properly.
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> That's a good point. This way, it is guaranteed that there is one def_ctx
> > > > > > >>>>>> per teedev. There should be a way to tell the open() callback that it is
> > > > > > >>>>>> a def_ctx, so it is not registered as a supplicant context.
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>> Should we allow this function to be called more than once for each teedev?
> > > > > > >>>>>>
> > > > > > >>>>>> Yes, moving to teedev_open() will fix the issue.
> > > > > > >>>>>>
> > > > > > >>>>>>> Do we need serialization in this function if it's called after the
> > > > > > >>>>>>> driver is probed?
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> True. I'll make sure there is no race.
> > > > > > >>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> -static void teedev_ctx_release(struct kref *ref)
> > > > > > >>>>>>>> -{
> > > > > > >>>>>>>> -       struct tee_context *ctx = container_of(ref, struct tee_context,
> > > > > > >>>>>>>> -                                              refcount);
> > > > > > >>>>>>>> -       ctx->releasing = true;
> > > > > > >>>>>>>> -       ctx->teedev->desc->ops->release(ctx);
> > > > > > >>>>>>>> -       kfree(ctx);
> > > > > > >>>>>>>> +       return ctx;
> > > > > > >>>>>>>>  }
> > > > > > >>>>>>>> +EXPORT_SYMBOL_GPL(teedev_get_def_context);
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> -void teedev_ctx_put(struct tee_context *ctx)
> > > > > > >>>>>>>> +void teedev_close_context(struct tee_context *ctx)
> > > > > > >>>>>>>>  {
> > > > > > >>>>>>>> -       if (ctx->releasing)
> > > > > > >>>>>>>> +       struct tee_device *teedev = ctx->teedev;
> > > > > > >>>>>>>> +       struct tee_shm *shm;
> > > > > > >>>>>>>> +
> > > > > > >>>>>>>> +       if (ctx == &teedev->def_ctx)
> > > > > > >>>>>>>>                 return;
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> -       kref_put(&ctx->refcount, teedev_ctx_release);
> > > > > > >>>>>>>> -}
> > > > > > >>>>>>>> +       teedev->desc->ops->release(ctx);
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> -void teedev_close_context(struct tee_context *ctx)
> > > > > > >>>>>>>> -{
> > > > > > >>>>>>>> -       struct tee_device *teedev = ctx->teedev;
> > > > > > >>>>>>>> +       mutex_lock(&teedev->mutex);
> > > > > > >>>>>>>> +       list_for_each_entry(shm, &ctx->list_shm, link) {
> > > > > > >>>>>>>> +               /* Context released. However, shm still holding a teedev reference.
> > > > > > >>>>>>>> +                * Replace shm->ctx with the default context so that tee_shm_get_from_id()
> > > > > > >>>>>>>> +                * fails (i.e. it is not accessible from userspace) but shm still
> > > > > > >>>>>>>> +                * holds a valid context for further clean up, e.g. shm_unregister().
> > > > > > >>>>>>>> +                */
> > > > > > >>>>>>>
> > > > > > >>>>>>> /*
> > > > > > >>>>>>>  * Please format
> > > > > > >>>>>>>  * multiline comments
> > > > > > >>>>>>>  * like this. Please
> > > > > > >>>>>>>  * keep the lines at
> > > > > > >>>>>>>  * max 80 columns
> > > > > > >>>>>>>  * here and at other
> > > > > > >>>>>>>  * places in the patch-
> > > > > > >>>>>>>  * set.
> > > > > > >>>>>>>  */
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> Ack.
> > > > > > >>>>>>
> > > > > > >>>>>>>> +               shm->ctx = &teedev->def_ctx;
> > > > > > >>>>>>>
> > > > > > >>>>>>> shm->ctx will always point to a valid context, even if it is the
> > > > > > >>>>>>> default context. It seems that we can always get hold of the correct
> > > > > > >>>>>>> teedev via shm->ctx->teedev. Do we need "tee: revert removal of
> > > > > > >>>>>>> redundant teedev in struct tee_shm"?
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> It was there in case we wanted to use NULL, but with def_ctx, it is not
> > > > > > >>>>>> necessary. I am withdrawing that commit. :).
> > > > > > >>>>>>
> > > > > > >>>>>>> Shouldn't the shm be removed from the ctx->list_shm and be moved to
> > > > > > >>>>>>> teedev->def_ctx.list_shm?
> > > > > > >>>>>
> > > > > > >>>>> +1
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>> Ack.
> > > > > > >>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> Not really. If we put shm in the teedev->def_ctx.list_shm, by the time
> > > > > > >>>>>> we are closing the def_ctx, the list is guaranteed to be empty.
> > > > > > >>>>>>
> > > > > > >>>>>> However, I understand it is cleaner and more consistent to do that rather
> > > > > > >>>>>> than making changes to tee_shm_put().
> > > > > > >>>>>>
> > > > > > >>>>>> I'll do it.
> > > > > > >>>>>>
> > > > > > >>>>>>>> +       }
> > > > > > >>>>>>>> +       mutex_unlock(&teedev->mutex);
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> -       teedev_ctx_put(ctx);
> > > > > > >>>>>>>> +       kfree(ctx);
> > > > > > >>>>>>>>         tee_device_put(teedev);
> > > > > > >>>>>>>>  }
> > > > > > >>>>>>>>  EXPORT_SYMBOL_GPL(teedev_close_context);
> > > > > > >>>>>>>> @@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>         teedev->desc = teedesc;
> > > > > > >>>>>>>>         teedev->pool = pool;
> > > > > > >>>>>>>> +       /* Only open default context when teedev_get_def_context() called. */
> > > > > > >>>>>>>> +       teedev->def_ctx.teedev = NULL;
> > > > > > >>>>>
> > > > > > >>>>> Why don't you open the device context here only? This will associate
> > > > > > >>>>> it automatically with teedev lifespan and then
> > > > > > >>>>> teedev_get_def_context() will just return a reference to that.
> > > > > > >>>>>
> > > > > > >>>>> -Sumit
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>> So my assumption is that the tee_devic_alloc() is called as part of
> > > > > > >>>> the driver initialization; there is no guarantee that at this time the
> > > > > > >>>> driver is actually ready to accept any open() callback.
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>> The drivers should be able to handle open() callback since we already
> > > > > > >>> check for !teedesc->ops->open in the beginning of tee_devic_alloc().
> > > > > > >>> Also, we need to open a device context for !TEE_DESC_PRIVILEGED such
> > > > > > >>> that we don't open a supplicant device context there.
> > > > > > >>
> > > > > > >> It would be nice to have the device context fully initialized when the
> > > > > > >> probe function returns. How about adding a "bool is_dev_ctx" to struct
> > > > > > >> tee_context so the open() callback can tell that this is a special
> > > > > > >> tee_contex?
> > > > > > >
> > > > > > > Sure, that will be useful to distinguish the device context from
> > > > > > > normal client context.
> > > > > > >
> > > > > > > -Sumit
> > > > > > >
> > > > > >
> > > > > > So, as far as the open() callback, I do not believe checking if it is not null
> > > > > > is reasonable for calling it here. Most drivers allocate resources and then
> > > > > > initialize them. So, assume these steps for a TEE driver:
> > > > > >  (1) allocate internal data structures,
> > > > > >  (2) allocate the device,
> > > > > >  (3) initialize the internal data structurse and then
> > > > > >  (4) register the device.
> > > > > >
> > > > > > Having these steps for a backend driver means that if you call open() at
> > > > > > step (2), the internal data structures are not ready.
> > > >
> > > > As part of tee_device_alloc(), every driver has to pass "const struct
> > > > tee_desc *teedesc" fully initialized. Which internal data structures
> > > > are you referring too? Is there any upstream example?
> > >
> > > It's reasonable to wait with the open() callback until step 4 above,
> > > which should correspond with the tee_device_register() call. Data
> > > written only once doesn't need serialized access if the fields are
> > > only accessed after they have been fully initialized.
> >
> > Fair enough, I can live with the device context opened after registering it.
> >
> > >
> > > >
> > > > > >
> > > > > > I was originally thinking of going with Jens' suggestion to open dev_ctx in
> > > > > > the teedev_open(), and use a flag to distinguish the type of context for
> > > > > > the open() callback
> > > > > >
> > > > > > What about this:
> > > > > > Open the dev_ctx in the tee_device_register(), at the last step before
> > > > > > setting the TEE_DEVICE_FLAG_REGISTERED flag. Then the open() callback can
> > > > > > check for this flag to determine if it is a normal context or dev_ctx.
> > > > > > If the open() is called while the device has not been registered, it should
> > > > > > handle it differently
> > > > >
> > > > > That makes sense, the driver should be prepared to handle open() calls
> > > > > after tee_device_register() anyway.
> > > > > However, there is no serialization of the flags field in struct
> > > > > tee_device. Hmm, would it be too hacky for the open() callback to
> > > > > check if &ctx->teedev.dev_ctx == ctx? We could add a helper function
> > > > > to wrap that check.
> > > > >
> > > >
> > > > Your suggested change requires every driver to update open() callback
> > > > and later other callbacks may have to support it too. IMHO, only
> > > > teedev_get_dev_ctx() should be able to return a reference to device
> > > > context for usage within the TEE and the implementation driver.
> > >
> > > Yes, but it's only the OP-TEE driver that needs anything special. It
> > > looks like the others can be left unchanged.
> >
> > I suppose it's most likely the upcoming QTEE driver requiring it.
> >
> > >
> > > >
> > > > I am still not able to understand why the following won't work with a
> > > > clear lifetime for the device context?
> > > >
> > > > tee_device_alloc()
> > > >   -> if (!(teedesc->flags & TEE_DESC_PRIVILEGED))
> > > > desc->ops->open(&teedev->dev_ctx);
> > >
> > > We must also have a fully initialized dev_ctx for the supplicant
> > > device.
> >
> > Currently I only see following for OP-TEE driver:
> >
> > ctx = teedev_open(optee->teedev);
> >
> > And I can't see anything like below:
> >
> > ctx = teedev_open(optee->supp_teedev);
> >
> > Where do you think that the dev_ctx is required for a supplicant
> > device? AFAICS, currently opening a context with the supplicant device
> > means that the supplicant daemon is available to handle RPCs which
> > won't be possible during OP-TEE driver probe. Am I missing something?
>
> One reason for initializing dev_ctx for all tee_devices is in
> teedev_close_context(), where the tee_shms still active are
> transferred to dev_ctx. The teedev member was re-introduced in this
> patch set, but it can be removed again if we can depend on the dev_ctx
> to always be available in teedev_close_context(). Even the
> tee-supplicant may close its tee_context with active tee_shms at some
> point. It might be possible to use half-baked dev_ctx, but then we'd
> be burdened with keeping track of which dev_ctx can be used for what.
> We want as few special cases as possible.
>

Okay that sounds reasonable to lower the complexity. However,
currently we only allow a single supplicant context so having another
device context exception for that shouldn't be an issue.

-Sumit


> >
> > > I'd rather delay the open() callback until
> > > tee_device_register() since the dev_ctx is guaranteed not to be needed
> > > before that.
> >
> > Okay, the updated call chain can look like:
> >
> > tee_device_register()
> >   -> if (!(teedev->desc->flags & TEE_DESC_PRIVILEGED))
> >          desc->ops->open(&teedev->dev_ctx);
> > >
> > > >
> > > > tee_device_put()
> > > >   -> if (teedev->dev_ctx) desc->ops->release(&teedev->dev_ctx);
> > >
> > > teedev->dev_ctx is supposed to be embedded in struct tee_device, so
> > > the if isn't needed.
> >
> > I added "if" to cover the case when dev_ctx is not initialized for the
> > supplicant device.
>
> OK.
>
> Cheers,
> Jens
>
> >
> > -Sumit
> >
> > >
> > > Cheers,
> > > Jens
> > >
> > > >
> > > > -Sumit
> > > >
> > > > > Cheers,
> > > > > Jens
> > > > >
> > > > > >
> > > > > > - Amir
> > > > > >
> > > > > > >>
> > > > > > >> Cheers,
> > > > > > >> Jens
> > > > > > >>
> > > > > > >>>
> > > > > > >>> -Sumit

Re: [PATCH RFC 3/3] tee: introduce orphan tee_shm and default context

[Amirreza Zarrabi]

On 11/27/2024 5:01 PM, Sumit Garg wrote:
> On Tue, 26 Nov 2024 at 20:52, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>
>> On Tue, Nov 26, 2024 at 1:27 PM Sumit Garg <sumit.garg@linaro.org> wrote:
>>>
>>> On Tue, 26 Nov 2024 at 14:03, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>>>
>>>> On Mon, Nov 25, 2024 at 9:55 PM Amirreza Zarrabi
>>>> <quic_azarrabi@quicinc.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 11/25/2024 6:51 PM, Sumit Garg wrote:
>>>>>> On Mon, 25 Nov 2024 at 12:53, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>>>>>>
>>>>>>> On Mon, Nov 25, 2024 at 7:14 AM Sumit Garg <sumit.garg@linaro.org> wrote:
>>>>>>>>
>>>>>>>> On Mon, 25 Nov 2024 at 03:00, Amirreza Zarrabi
>>>>>>>> <quic_azarrabi@quicinc.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Sumit,
>>>>>>>>>
>>>>>>>>> Thank you so much for the comemnts :).
>>>>>>>>>
>>>>>>>>> On 11/23/2024 9:32 PM, Sumit Garg wrote:
>>>>>>>>>> Hi Amirreza,
>>>>>>>>>>
>>>>>>>>>> Thanks for proposing this.
>>>>>>>>>>
>>>>>>>>>> On Fri, 22 Nov 2024 at 06:38, Amirreza Zarrabi
>>>>>>>>>> <quic_azarrabi@quicinc.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 11/21/2024 11:08 PM, Jens Wiklander wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Jens,
>>>>>>>>>>>
>>>>>>>>>>>> Hi Amirreza,
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Nov 21, 2024 at 2:37 AM Amirreza Zarrabi
>>>>>>>>>>>> <quic_azarrabi@quicinc.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> The default context has a lifespan similar to the tee_device.
>>>>>>>>>>
>>>>>>>>>> Since it's associated with tee_device context, let's call it obvious
>>>>>>>>>> via renaming it as device context instead (s/def_ctx/dev_ctx/ in this
>>>>>>>>>> patch).
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Make sense, I'll rename it.
>>>>>>>>>
>>>>>>>>>>>>> It is used as a context for shared memory if the context to which the
>>>>>>>>>>>>> shared memory belongs is released, making the tee_shm an orphan.
>>>>>>>>>>>>> This allows the driver implementing shm_unregister to safely make
>>>>>>>>>>>>> subsequent calls, such as to a supplicant if needed.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It also enables users to free the shared memory while the driver is
>>>>>>>>>>>>> blocked on unregister_tee_device safely.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Preferably, this should be used for all driver internal uses, using
>>>>>>>>>>>>> teedev_get_def_context rather than calling teedev_open.
>>>>>>>>>>
>>>>>>>>>> Makes sense to me.
>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
>>>>>>>>>>>>> ---
>>>>>>>>>>>>>  drivers/tee/optee/core.c    |  2 +-
>>>>>>>>>>>>>  drivers/tee/optee/ffa_abi.c |  2 +-
>>>>>>>>>>>>>  drivers/tee/optee/smc_abi.c |  2 +-
>>>>>>>>>>>>>  drivers/tee/tee_core.c      | 83 +++++++++++++++++++++++++++++----------------
>>>>>>>>>>>>>  drivers/tee/tee_private.h   |  3 --
>>>>>>>>>>>>>  drivers/tee/tee_shm.c       | 18 ++--------
>>>>>>>>>>>>>  include/linux/tee_core.h    | 15 ++++++++
>>>>>>>>>>>>>  include/linux/tee_drv.h     |  7 ----
>>>>>>>>>>>>>  8 files changed, 73 insertions(+), 59 deletions(-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
>>>>>>>>>>>>> index c75fddc83576..78d43d0c8014 100644
>>>>>>>>>>>>> --- a/drivers/tee/optee/core.c
>>>>>>>>>>>>> +++ b/drivers/tee/optee/core.c
>>>>>>>>>>>>> @@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
>>>>>>>>>>>>>
>>>>>>>>>>>>>         optee_notif_uninit(optee);
>>>>>>>>>>>>>         optee_shm_arg_cache_uninit(optee);
>>>>>>>>>>>>> -       teedev_close_context(optee->ctx);
>>>>>>>>>>>>> +
>>>>>>>>>>>>>         /*
>>>>>>>>>>>>>          * The two devices have to be unregistered before we can free the
>>>>>>>>>>>>>          * other resources.
>>>>>>>>>>>>> diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
>>>>>>>>>>>>> index f3af5666bb11..6ad94f0788ad 100644
>>>>>>>>>>>>> --- a/drivers/tee/optee/ffa_abi.c
>>>>>>>>>>>>> +++ b/drivers/tee/optee/ffa_abi.c
>>>>>>>>>>>>> @@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
>>>>>>>>>>>>>         optee_shm_arg_cache_init(optee, arg_cache_flags);
>>>>>>>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
>>>>>>>>>>>>>         ffa_dev_set_drvdata(ffa_dev, optee);
>>>>>>>>>>>>> -       ctx = teedev_open(optee->teedev);
>>>>>>>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
>>>>>>>>>>>>>         if (IS_ERR(ctx)) {
>>>>>>>>>>>>>                 rc = PTR_ERR(ctx);
>>>>>>>>>>>>>                 goto err_rhashtable_free;
>>>>>>>>>>>>> diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
>>>>>>>>>>>>> index e9456e3e74cc..c77a3e631d04 100644
>>>>>>>>>>>>> --- a/drivers/tee/optee/smc_abi.c
>>>>>>>>>>>>> +++ b/drivers/tee/optee/smc_abi.c
>>>>>>>>>>>>> @@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
>>>>>>>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
>>>>>>>>>>>>>
>>>>>>>>>>>>>         platform_set_drvdata(pdev, optee);
>>>>>>>>>>>>> -       ctx = teedev_open(optee->teedev);
>>>>>>>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
>>>>>>>>>>>>>         if (IS_ERR(ctx)) {
>>>>>>>>>>>>>                 rc = PTR_ERR(ctx);
>>>>>>>>>>>>>                 goto err_supp_uninit;
>>>>>>>>>>>>> diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
>>>>>>>>>>>>> index 93f3b330aec8..805e1336089d 100644
>>>>>>>>>>>>> --- a/drivers/tee/tee_core.c
>>>>>>>>>>>>> +++ b/drivers/tee/tee_core.c
>>>>>>>>>>>>> @@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
>>>>>>>>>>>>>                 goto err;
>>>>>>>>>>>>>         }
>>>>>>>>>>>>>
>>>>>>>>>>>>> -       kref_init(&ctx->refcount);
>>>>>>>>>>>>>         ctx->teedev = teedev;
>>>>>>>>>>>>>         INIT_LIST_HEAD(&ctx->list_shm);
>>>>>>>>>>>>>         rc = teedev->desc->ops->open(ctx);
>>>>>>>>>>>>> @@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
>>>>>>>>>>>>>  }
>>>>>>>>>>>>>  EXPORT_SYMBOL_GPL(teedev_open);
>>>>>>>>>>>>>
>>>>>>>>>>>>> -void teedev_ctx_get(struct tee_context *ctx)
>>>>>>>>>>>>> +struct tee_context *teedev_get_def_context(struct tee_device *teedev)
>>>>>>>>>>>>>  {
>>>>>>>>>>>>> -       if (ctx->releasing)
>>>>>>>>>>>>> -               return;
>>>>>>>>>>>>> +       int rc;
>>>>>>>>>>>>> +       struct tee_context *ctx = &teedev->def_ctx;
>>>>>>>>>>>>>
>>>>>>>>>>>>> -       kref_get(&ctx->refcount);
>>>>>>>>>>>>> -}
>>>>>>>>>>>>> +       ctx->teedev = teedev;
>>>>>>>>>>>>> +       INIT_LIST_HEAD(&ctx->list_shm);
>>>>>>>>>>>>> +       rc = teedev->desc->ops->open(ctx);
>>>>>>>>>>>>> +       if (rc)
>>>>>>>>>>>>> +               return ERR_PTR(rc);
>>>>>>>>>>>>
>>>>>>>>>>>> I think ctx->teedev and ctx->list_shm must always be initialized or
>>>>>>>>>>>> &teedev->def_ctx can't be used in teedev_close_context().
>>>>>>>>>>>
>>>>>>>>>>> True, but &teedev->def_ctx is never used in teedev_close_context().
>>>>>>>>>>> The closing of the &teedev->def_ctx simply ignored. So once opened,
>>>>>>>>>>> &teedev->def_ctx will always remain open until the tee_device is alive.
>>>>>>>>>>>
>>>>>>>>>>>> We could initialize teedev->def_ctx on the first call to teedev_open()
>>>>>>>>>>>> on that tee_device. We need a way to tell the
>>>>>>>>>>>> teedev->desc->ops->open() to the backed driver that it's initializing
>>>>>>>>>>>> the default context though, or optee_open() can't handle the
>>>>>>>>>>>> tee-supplicant case properly.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> That's a good point. This way, it is guaranteed that there is one def_ctx
>>>>>>>>>>> per teedev. There should be a way to tell the open() callback that it is
>>>>>>>>>>> a def_ctx, so it is not registered as a supplicant context.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Should we allow this function to be called more than once for each teedev?
>>>>>>>>>>>
>>>>>>>>>>> Yes, moving to teedev_open() will fix the issue.
>>>>>>>>>>>
>>>>>>>>>>>> Do we need serialization in this function if it's called after the
>>>>>>>>>>>> driver is probed?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> True. I'll make sure there is no race.
>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -static void teedev_ctx_release(struct kref *ref)
>>>>>>>>>>>>> -{
>>>>>>>>>>>>> -       struct tee_context *ctx = container_of(ref, struct tee_context,
>>>>>>>>>>>>> -                                              refcount);
>>>>>>>>>>>>> -       ctx->releasing = true;
>>>>>>>>>>>>> -       ctx->teedev->desc->ops->release(ctx);
>>>>>>>>>>>>> -       kfree(ctx);
>>>>>>>>>>>>> +       return ctx;
>>>>>>>>>>>>>  }
>>>>>>>>>>>>> +EXPORT_SYMBOL_GPL(teedev_get_def_context);
>>>>>>>>>>>>>
>>>>>>>>>>>>> -void teedev_ctx_put(struct tee_context *ctx)
>>>>>>>>>>>>> +void teedev_close_context(struct tee_context *ctx)
>>>>>>>>>>>>>  {
>>>>>>>>>>>>> -       if (ctx->releasing)
>>>>>>>>>>>>> +       struct tee_device *teedev = ctx->teedev;
>>>>>>>>>>>>> +       struct tee_shm *shm;
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +       if (ctx == &teedev->def_ctx)
>>>>>>>>>>>>>                 return;
>>>>>>>>>>>>>
>>>>>>>>>>>>> -       kref_put(&ctx->refcount, teedev_ctx_release);
>>>>>>>>>>>>> -}
>>>>>>>>>>>>> +       teedev->desc->ops->release(ctx);
>>>>>>>>>>>>>
>>>>>>>>>>>>> -void teedev_close_context(struct tee_context *ctx)
>>>>>>>>>>>>> -{
>>>>>>>>>>>>> -       struct tee_device *teedev = ctx->teedev;
>>>>>>>>>>>>> +       mutex_lock(&teedev->mutex);
>>>>>>>>>>>>> +       list_for_each_entry(shm, &ctx->list_shm, link) {
>>>>>>>>>>>>> +               /* Context released. However, shm still holding a teedev reference.
>>>>>>>>>>>>> +                * Replace shm->ctx with the default context so that tee_shm_get_from_id()
>>>>>>>>>>>>> +                * fails (i.e. it is not accessible from userspace) but shm still
>>>>>>>>>>>>> +                * holds a valid context for further clean up, e.g. shm_unregister().
>>>>>>>>>>>>> +                */
>>>>>>>>>>>>
>>>>>>>>>>>> /*
>>>>>>>>>>>>  * Please format
>>>>>>>>>>>>  * multiline comments
>>>>>>>>>>>>  * like this. Please
>>>>>>>>>>>>  * keep the lines at
>>>>>>>>>>>>  * max 80 columns
>>>>>>>>>>>>  * here and at other
>>>>>>>>>>>>  * places in the patch-
>>>>>>>>>>>>  * set.
>>>>>>>>>>>>  */
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Ack.
>>>>>>>>>>>
>>>>>>>>>>>>> +               shm->ctx = &teedev->def_ctx;
>>>>>>>>>>>>
>>>>>>>>>>>> shm->ctx will always point to a valid context, even if it is the
>>>>>>>>>>>> default context. It seems that we can always get hold of the correct
>>>>>>>>>>>> teedev via shm->ctx->teedev. Do we need "tee: revert removal of
>>>>>>>>>>>> redundant teedev in struct tee_shm"?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> It was there in case we wanted to use NULL, but with def_ctx, it is not
>>>>>>>>>>> necessary. I am withdrawing that commit. :).
>>>>>>>>>>>
>>>>>>>>>>>> Shouldn't the shm be removed from the ctx->list_shm and be moved to
>>>>>>>>>>>> teedev->def_ctx.list_shm?
>>>>>>>>>>
>>>>>>>>>> +1
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ack.
>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Not really. If we put shm in the teedev->def_ctx.list_shm, by the time
>>>>>>>>>>> we are closing the def_ctx, the list is guaranteed to be empty.
>>>>>>>>>>>
>>>>>>>>>>> However, I understand it is cleaner and more consistent to do that rather
>>>>>>>>>>> than making changes to tee_shm_put().
>>>>>>>>>>>
>>>>>>>>>>> I'll do it.
>>>>>>>>>>>
>>>>>>>>>>>>> +       }
>>>>>>>>>>>>> +       mutex_unlock(&teedev->mutex);
>>>>>>>>>>>>>
>>>>>>>>>>>>> -       teedev_ctx_put(ctx);
>>>>>>>>>>>>> +       kfree(ctx);
>>>>>>>>>>>>>         tee_device_put(teedev);
>>>>>>>>>>>>>  }
>>>>>>>>>>>>>  EXPORT_SYMBOL_GPL(teedev_close_context);
>>>>>>>>>>>>> @@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
>>>>>>>>>>>>>
>>>>>>>>>>>>>         teedev->desc = teedesc;
>>>>>>>>>>>>>         teedev->pool = pool;
>>>>>>>>>>>>> +       /* Only open default context when teedev_get_def_context() called. */
>>>>>>>>>>>>> +       teedev->def_ctx.teedev = NULL;
>>>>>>>>>>
>>>>>>>>>> Why don't you open the device context here only? This will associate
>>>>>>>>>> it automatically with teedev lifespan and then
>>>>>>>>>> teedev_get_def_context() will just return a reference to that.
>>>>>>>>>>
>>>>>>>>>> -Sumit
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So my assumption is that the tee_devic_alloc() is called as part of
>>>>>>>>> the driver initialization; there is no guarantee that at this time the
>>>>>>>>> driver is actually ready to accept any open() callback.
>>>>>>>>>
>>>>>>>>
>>>>>>>> The drivers should be able to handle open() callback since we already
>>>>>>>> check for !teedesc->ops->open in the beginning of tee_devic_alloc().
>>>>>>>> Also, we need to open a device context for !TEE_DESC_PRIVILEGED such
>>>>>>>> that we don't open a supplicant device context there.
>>>>>>>
>>>>>>> It would be nice to have the device context fully initialized when the
>>>>>>> probe function returns. How about adding a "bool is_dev_ctx" to struct
>>>>>>> tee_context so the open() callback can tell that this is a special
>>>>>>> tee_contex?
>>>>>>
>>>>>> Sure, that will be useful to distinguish the device context from
>>>>>> normal client context.
>>>>>>
>>>>>> -Sumit
>>>>>>
>>>>>
>>>>> So, as far as the open() callback, I do not believe checking if it is not null
>>>>> is reasonable for calling it here. Most drivers allocate resources and then
>>>>> initialize them. So, assume these steps for a TEE driver:
>>>>>  (1) allocate internal data structures,
>>>>>  (2) allocate the device,
>>>>>  (3) initialize the internal data structurse and then
>>>>>  (4) register the device.
>>>>>
>>>>> Having these steps for a backend driver means that if you call open() at
>>>>> step (2), the internal data structures are not ready.
>>>
>>> As part of tee_device_alloc(), every driver has to pass "const struct
>>> tee_desc *teedesc" fully initialized. Which internal data structures
>>> are you referring too? Is there any upstream example?
>>
>> It's reasonable to wait with the open() callback until step 4 above,
>> which should correspond with the tee_device_register() call. Data
>> written only once doesn't need serialized access if the fields are
>> only accessed after they have been fully initialized.
> 
> Fair enough, I can live with the device context opened after registering it.
> 
>>
>>>
>>>>>
>>>>> I was originally thinking of going with Jens' suggestion to open dev_ctx in
>>>>> the teedev_open(), and use a flag to distinguish the type of context for
>>>>> the open() callback
>>>>>
>>>>> What about this:
>>>>> Open the dev_ctx in the tee_device_register(), at the last step before
>>>>> setting the TEE_DEVICE_FLAG_REGISTERED flag. Then the open() callback can
>>>>> check for this flag to determine if it is a normal context or dev_ctx.
>>>>> If the open() is called while the device has not been registered, it should
>>>>> handle it differently
>>>>
>>>> That makes sense, the driver should be prepared to handle open() calls
>>>> after tee_device_register() anyway.
>>>> However, there is no serialization of the flags field in struct
>>>> tee_device. Hmm, would it be too hacky for the open() callback to
>>>> check if &ctx->teedev.dev_ctx == ctx? We could add a helper function
>>>> to wrap that check.
>>>>
>>>
>>> Your suggested change requires every driver to update open() callback
>>> and later other callbacks may have to support it too. IMHO, only
>>> teedev_get_dev_ctx() should be able to return a reference to device
>>> context for usage within the TEE and the implementation driver.
>>
>> Yes, but it's only the OP-TEE driver that needs anything special. It
>> looks like the others can be left unchanged.
> 
> I suppose it's most likely the upcoming QTEE driver requiring it.
>

I don't believe this is correct. This requirement is implicitly imposed
by the TEE subsystem API. If calling open() is acceptable in
tee_device_alloc(), then I could argue that tee_device_register() and
tee_device_alloc() should be merged into a single function. If a driver
is ready to handle requests, why delay its exposure by postponing the
registration?

By calling open() in tee_device_alloc(), you indirectly impose an unspoken
requirement on developers regarding how they should write their drivers,
such as the steps they should take to probe the device.

Regards,
Amir

>>
>>>
>>> I am still not able to understand why the following won't work with a
>>> clear lifetime for the device context?
>>>
>>> tee_device_alloc()
>>>   -> if (!(teedesc->flags & TEE_DESC_PRIVILEGED))
>>> desc->ops->open(&teedev->dev_ctx);
>>
>> We must also have a fully initialized dev_ctx for the supplicant
>> device.
> 
> Currently I only see following for OP-TEE driver:
> 
> ctx = teedev_open(optee->teedev);
> 
> And I can't see anything like below:
> 
> ctx = teedev_open(optee->supp_teedev);
> 
> Where do you think that the dev_ctx is required for a supplicant
> device? AFAICS, currently opening a context with the supplicant device
> means that the supplicant daemon is available to handle RPCs which
> won't be possible during OP-TEE driver probe. Am I missing something?
> 
>> I'd rather delay the open() callback until
>> tee_device_register() since the dev_ctx is guaranteed not to be needed
>> before that.
> 
> Okay, the updated call chain can look like:
> 
> tee_device_register()
>   -> if (!(teedev->desc->flags & TEE_DESC_PRIVILEGED))
>          desc->ops->open(&teedev->dev_ctx);
>>
>>>
>>> tee_device_put()
>>>   -> if (teedev->dev_ctx) desc->ops->release(&teedev->dev_ctx);
>>
>> teedev->dev_ctx is supposed to be embedded in struct tee_device, so
>> the if isn't needed.
> 
> I added "if" to cover the case when dev_ctx is not initialized for the
> supplicant device.
> 
> -Sumit
> 
>>
>> Cheers,
>> Jens
>>
>>>
>>> -Sumit
>>>
>>>> Cheers,
>>>> Jens
>>>>
>>>>>
>>>>> - Amir
>>>>>
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Jens
>>>>>>>
>>>>>>>>
>>>>>>>> -Sumit

Re: [PATCH RFC 3/3] tee: introduce orphan tee_shm and default context

[Sumit Garg]

On Thu, 28 Nov 2024 at 02:29, Amirreza Zarrabi
<quic_azarrabi@quicinc.com> wrote:
>
>
>
> On 11/27/2024 5:01 PM, Sumit Garg wrote:
> > On Tue, 26 Nov 2024 at 20:52, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >>
> >> On Tue, Nov 26, 2024 at 1:27 PM Sumit Garg <sumit.garg@linaro.org> wrote:
> >>>
> >>> On Tue, 26 Nov 2024 at 14:03, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >>>>
> >>>> On Mon, Nov 25, 2024 at 9:55 PM Amirreza Zarrabi
> >>>> <quic_azarrabi@quicinc.com> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 11/25/2024 6:51 PM, Sumit Garg wrote:
> >>>>>> On Mon, 25 Nov 2024 at 12:53, Jens Wiklander <jens.wiklander@linaro.org> wrote:
> >>>>>>>
> >>>>>>> On Mon, Nov 25, 2024 at 7:14 AM Sumit Garg <sumit.garg@linaro.org> wrote:
> >>>>>>>>
> >>>>>>>> On Mon, 25 Nov 2024 at 03:00, Amirreza Zarrabi
> >>>>>>>> <quic_azarrabi@quicinc.com> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Hi Sumit,
> >>>>>>>>>
> >>>>>>>>> Thank you so much for the comemnts :).
> >>>>>>>>>
> >>>>>>>>> On 11/23/2024 9:32 PM, Sumit Garg wrote:
> >>>>>>>>>> Hi Amirreza,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for proposing this.
> >>>>>>>>>>
> >>>>>>>>>> On Fri, 22 Nov 2024 at 06:38, Amirreza Zarrabi
> >>>>>>>>>> <quic_azarrabi@quicinc.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On 11/21/2024 11:08 PM, Jens Wiklander wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Jens,
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi Amirreza,
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Nov 21, 2024 at 2:37 AM Amirreza Zarrabi
> >>>>>>>>>>>> <quic_azarrabi@quicinc.com> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The default context has a lifespan similar to the tee_device.
> >>>>>>>>>>
> >>>>>>>>>> Since it's associated with tee_device context, let's call it obvious
> >>>>>>>>>> via renaming it as device context instead (s/def_ctx/dev_ctx/ in this
> >>>>>>>>>> patch).
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Make sense, I'll rename it.
> >>>>>>>>>
> >>>>>>>>>>>>> It is used as a context for shared memory if the context to which the
> >>>>>>>>>>>>> shared memory belongs is released, making the tee_shm an orphan.
> >>>>>>>>>>>>> This allows the driver implementing shm_unregister to safely make
> >>>>>>>>>>>>> subsequent calls, such as to a supplicant if needed.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> It also enables users to free the shared memory while the driver is
> >>>>>>>>>>>>> blocked on unregister_tee_device safely.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Preferably, this should be used for all driver internal uses, using
> >>>>>>>>>>>>> teedev_get_def_context rather than calling teedev_open.
> >>>>>>>>>>
> >>>>>>>>>> Makes sense to me.
> >>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
> >>>>>>>>>>>>> ---
> >>>>>>>>>>>>>  drivers/tee/optee/core.c    |  2 +-
> >>>>>>>>>>>>>  drivers/tee/optee/ffa_abi.c |  2 +-
> >>>>>>>>>>>>>  drivers/tee/optee/smc_abi.c |  2 +-
> >>>>>>>>>>>>>  drivers/tee/tee_core.c      | 83 +++++++++++++++++++++++++++++----------------
> >>>>>>>>>>>>>  drivers/tee/tee_private.h   |  3 --
> >>>>>>>>>>>>>  drivers/tee/tee_shm.c       | 18 ++--------
> >>>>>>>>>>>>>  include/linux/tee_core.h    | 15 ++++++++
> >>>>>>>>>>>>>  include/linux/tee_drv.h     |  7 ----
> >>>>>>>>>>>>>  8 files changed, 73 insertions(+), 59 deletions(-)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
> >>>>>>>>>>>>> index c75fddc83576..78d43d0c8014 100644
> >>>>>>>>>>>>> --- a/drivers/tee/optee/core.c
> >>>>>>>>>>>>> +++ b/drivers/tee/optee/core.c
> >>>>>>>>>>>>> @@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>         optee_notif_uninit(optee);
> >>>>>>>>>>>>>         optee_shm_arg_cache_uninit(optee);
> >>>>>>>>>>>>> -       teedev_close_context(optee->ctx);
> >>>>>>>>>>>>> +
> >>>>>>>>>>>>>         /*
> >>>>>>>>>>>>>          * The two devices have to be unregistered before we can free the
> >>>>>>>>>>>>>          * other resources.
> >>>>>>>>>>>>> diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
> >>>>>>>>>>>>> index f3af5666bb11..6ad94f0788ad 100644
> >>>>>>>>>>>>> --- a/drivers/tee/optee/ffa_abi.c
> >>>>>>>>>>>>> +++ b/drivers/tee/optee/ffa_abi.c
> >>>>>>>>>>>>> @@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
> >>>>>>>>>>>>>         optee_shm_arg_cache_init(optee, arg_cache_flags);
> >>>>>>>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
> >>>>>>>>>>>>>         ffa_dev_set_drvdata(ffa_dev, optee);
> >>>>>>>>>>>>> -       ctx = teedev_open(optee->teedev);
> >>>>>>>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
> >>>>>>>>>>>>>         if (IS_ERR(ctx)) {
> >>>>>>>>>>>>>                 rc = PTR_ERR(ctx);
> >>>>>>>>>>>>>                 goto err_rhashtable_free;
> >>>>>>>>>>>>> diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
> >>>>>>>>>>>>> index e9456e3e74cc..c77a3e631d04 100644
> >>>>>>>>>>>>> --- a/drivers/tee/optee/smc_abi.c
> >>>>>>>>>>>>> +++ b/drivers/tee/optee/smc_abi.c
> >>>>>>>>>>>>> @@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
> >>>>>>>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>         platform_set_drvdata(pdev, optee);
> >>>>>>>>>>>>> -       ctx = teedev_open(optee->teedev);
> >>>>>>>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
> >>>>>>>>>>>>>         if (IS_ERR(ctx)) {
> >>>>>>>>>>>>>                 rc = PTR_ERR(ctx);
> >>>>>>>>>>>>>                 goto err_supp_uninit;
> >>>>>>>>>>>>> diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
> >>>>>>>>>>>>> index 93f3b330aec8..805e1336089d 100644
> >>>>>>>>>>>>> --- a/drivers/tee/tee_core.c
> >>>>>>>>>>>>> +++ b/drivers/tee/tee_core.c
> >>>>>>>>>>>>> @@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> >>>>>>>>>>>>>                 goto err;
> >>>>>>>>>>>>>         }
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -       kref_init(&ctx->refcount);
> >>>>>>>>>>>>>         ctx->teedev = teedev;
> >>>>>>>>>>>>>         INIT_LIST_HEAD(&ctx->list_shm);
> >>>>>>>>>>>>>         rc = teedev->desc->ops->open(ctx);
> >>>>>>>>>>>>> @@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
> >>>>>>>>>>>>>  }
> >>>>>>>>>>>>>  EXPORT_SYMBOL_GPL(teedev_open);
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -void teedev_ctx_get(struct tee_context *ctx)
> >>>>>>>>>>>>> +struct tee_context *teedev_get_def_context(struct tee_device *teedev)
> >>>>>>>>>>>>>  {
> >>>>>>>>>>>>> -       if (ctx->releasing)
> >>>>>>>>>>>>> -               return;
> >>>>>>>>>>>>> +       int rc;
> >>>>>>>>>>>>> +       struct tee_context *ctx = &teedev->def_ctx;
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -       kref_get(&ctx->refcount);
> >>>>>>>>>>>>> -}
> >>>>>>>>>>>>> +       ctx->teedev = teedev;
> >>>>>>>>>>>>> +       INIT_LIST_HEAD(&ctx->list_shm);
> >>>>>>>>>>>>> +       rc = teedev->desc->ops->open(ctx);
> >>>>>>>>>>>>> +       if (rc)
> >>>>>>>>>>>>> +               return ERR_PTR(rc);
> >>>>>>>>>>>>
> >>>>>>>>>>>> I think ctx->teedev and ctx->list_shm must always be initialized or
> >>>>>>>>>>>> &teedev->def_ctx can't be used in teedev_close_context().
> >>>>>>>>>>>
> >>>>>>>>>>> True, but &teedev->def_ctx is never used in teedev_close_context().
> >>>>>>>>>>> The closing of the &teedev->def_ctx simply ignored. So once opened,
> >>>>>>>>>>> &teedev->def_ctx will always remain open until the tee_device is alive.
> >>>>>>>>>>>
> >>>>>>>>>>>> We could initialize teedev->def_ctx on the first call to teedev_open()
> >>>>>>>>>>>> on that tee_device. We need a way to tell the
> >>>>>>>>>>>> teedev->desc->ops->open() to the backed driver that it's initializing
> >>>>>>>>>>>> the default context though, or optee_open() can't handle the
> >>>>>>>>>>>> tee-supplicant case properly.
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> That's a good point. This way, it is guaranteed that there is one def_ctx
> >>>>>>>>>>> per teedev. There should be a way to tell the open() callback that it is
> >>>>>>>>>>> a def_ctx, so it is not registered as a supplicant context.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> Should we allow this function to be called more than once for each teedev?
> >>>>>>>>>>>
> >>>>>>>>>>> Yes, moving to teedev_open() will fix the issue.
> >>>>>>>>>>>
> >>>>>>>>>>>> Do we need serialization in this function if it's called after the
> >>>>>>>>>>>> driver is probed?
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> True. I'll make sure there is no race.
> >>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -static void teedev_ctx_release(struct kref *ref)
> >>>>>>>>>>>>> -{
> >>>>>>>>>>>>> -       struct tee_context *ctx = container_of(ref, struct tee_context,
> >>>>>>>>>>>>> -                                              refcount);
> >>>>>>>>>>>>> -       ctx->releasing = true;
> >>>>>>>>>>>>> -       ctx->teedev->desc->ops->release(ctx);
> >>>>>>>>>>>>> -       kfree(ctx);
> >>>>>>>>>>>>> +       return ctx;
> >>>>>>>>>>>>>  }
> >>>>>>>>>>>>> +EXPORT_SYMBOL_GPL(teedev_get_def_context);
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -void teedev_ctx_put(struct tee_context *ctx)
> >>>>>>>>>>>>> +void teedev_close_context(struct tee_context *ctx)
> >>>>>>>>>>>>>  {
> >>>>>>>>>>>>> -       if (ctx->releasing)
> >>>>>>>>>>>>> +       struct tee_device *teedev = ctx->teedev;
> >>>>>>>>>>>>> +       struct tee_shm *shm;
> >>>>>>>>>>>>> +
> >>>>>>>>>>>>> +       if (ctx == &teedev->def_ctx)
> >>>>>>>>>>>>>                 return;
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -       kref_put(&ctx->refcount, teedev_ctx_release);
> >>>>>>>>>>>>> -}
> >>>>>>>>>>>>> +       teedev->desc->ops->release(ctx);
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -void teedev_close_context(struct tee_context *ctx)
> >>>>>>>>>>>>> -{
> >>>>>>>>>>>>> -       struct tee_device *teedev = ctx->teedev;
> >>>>>>>>>>>>> +       mutex_lock(&teedev->mutex);
> >>>>>>>>>>>>> +       list_for_each_entry(shm, &ctx->list_shm, link) {
> >>>>>>>>>>>>> +               /* Context released. However, shm still holding a teedev reference.
> >>>>>>>>>>>>> +                * Replace shm->ctx with the default context so that tee_shm_get_from_id()
> >>>>>>>>>>>>> +                * fails (i.e. it is not accessible from userspace) but shm still
> >>>>>>>>>>>>> +                * holds a valid context for further clean up, e.g. shm_unregister().
> >>>>>>>>>>>>> +                */
> >>>>>>>>>>>>
> >>>>>>>>>>>> /*
> >>>>>>>>>>>>  * Please format
> >>>>>>>>>>>>  * multiline comments
> >>>>>>>>>>>>  * like this. Please
> >>>>>>>>>>>>  * keep the lines at
> >>>>>>>>>>>>  * max 80 columns
> >>>>>>>>>>>>  * here and at other
> >>>>>>>>>>>>  * places in the patch-
> >>>>>>>>>>>>  * set.
> >>>>>>>>>>>>  */
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Ack.
> >>>>>>>>>>>
> >>>>>>>>>>>>> +               shm->ctx = &teedev->def_ctx;
> >>>>>>>>>>>>
> >>>>>>>>>>>> shm->ctx will always point to a valid context, even if it is the
> >>>>>>>>>>>> default context. It seems that we can always get hold of the correct
> >>>>>>>>>>>> teedev via shm->ctx->teedev. Do we need "tee: revert removal of
> >>>>>>>>>>>> redundant teedev in struct tee_shm"?
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> It was there in case we wanted to use NULL, but with def_ctx, it is not
> >>>>>>>>>>> necessary. I am withdrawing that commit. :).
> >>>>>>>>>>>
> >>>>>>>>>>>> Shouldn't the shm be removed from the ctx->list_shm and be moved to
> >>>>>>>>>>>> teedev->def_ctx.list_shm?
> >>>>>>>>>>
> >>>>>>>>>> +1
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Ack.
> >>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Not really. If we put shm in the teedev->def_ctx.list_shm, by the time
> >>>>>>>>>>> we are closing the def_ctx, the list is guaranteed to be empty.
> >>>>>>>>>>>
> >>>>>>>>>>> However, I understand it is cleaner and more consistent to do that rather
> >>>>>>>>>>> than making changes to tee_shm_put().
> >>>>>>>>>>>
> >>>>>>>>>>> I'll do it.
> >>>>>>>>>>>
> >>>>>>>>>>>>> +       }
> >>>>>>>>>>>>> +       mutex_unlock(&teedev->mutex);
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -       teedev_ctx_put(ctx);
> >>>>>>>>>>>>> +       kfree(ctx);
> >>>>>>>>>>>>>         tee_device_put(teedev);
> >>>>>>>>>>>>>  }
> >>>>>>>>>>>>>  EXPORT_SYMBOL_GPL(teedev_close_context);
> >>>>>>>>>>>>> @@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>         teedev->desc = teedesc;
> >>>>>>>>>>>>>         teedev->pool = pool;
> >>>>>>>>>>>>> +       /* Only open default context when teedev_get_def_context() called. */
> >>>>>>>>>>>>> +       teedev->def_ctx.teedev = NULL;
> >>>>>>>>>>
> >>>>>>>>>> Why don't you open the device context here only? This will associate
> >>>>>>>>>> it automatically with teedev lifespan and then
> >>>>>>>>>> teedev_get_def_context() will just return a reference to that.
> >>>>>>>>>>
> >>>>>>>>>> -Sumit
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> So my assumption is that the tee_devic_alloc() is called as part of
> >>>>>>>>> the driver initialization; there is no guarantee that at this time the
> >>>>>>>>> driver is actually ready to accept any open() callback.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> The drivers should be able to handle open() callback since we already
> >>>>>>>> check for !teedesc->ops->open in the beginning of tee_devic_alloc().
> >>>>>>>> Also, we need to open a device context for !TEE_DESC_PRIVILEGED such
> >>>>>>>> that we don't open a supplicant device context there.
> >>>>>>>
> >>>>>>> It would be nice to have the device context fully initialized when the
> >>>>>>> probe function returns. How about adding a "bool is_dev_ctx" to struct
> >>>>>>> tee_context so the open() callback can tell that this is a special
> >>>>>>> tee_contex?
> >>>>>>
> >>>>>> Sure, that will be useful to distinguish the device context from
> >>>>>> normal client context.
> >>>>>>
> >>>>>> -Sumit
> >>>>>>
> >>>>>
> >>>>> So, as far as the open() callback, I do not believe checking if it is not null
> >>>>> is reasonable for calling it here. Most drivers allocate resources and then
> >>>>> initialize them. So, assume these steps for a TEE driver:
> >>>>>  (1) allocate internal data structures,
> >>>>>  (2) allocate the device,
> >>>>>  (3) initialize the internal data structurse and then
> >>>>>  (4) register the device.
> >>>>>
> >>>>> Having these steps for a backend driver means that if you call open() at
> >>>>> step (2), the internal data structures are not ready.
> >>>
> >>> As part of tee_device_alloc(), every driver has to pass "const struct
> >>> tee_desc *teedesc" fully initialized. Which internal data structures
> >>> are you referring too? Is there any upstream example?
> >>
> >> It's reasonable to wait with the open() callback until step 4 above,
> >> which should correspond with the tee_device_register() call. Data
> >> written only once doesn't need serialized access if the fields are
> >> only accessed after they have been fully initialized.
> >
> > Fair enough, I can live with the device context opened after registering it.
> >
> >>
> >>>
> >>>>>
> >>>>> I was originally thinking of going with Jens' suggestion to open dev_ctx in
> >>>>> the teedev_open(), and use a flag to distinguish the type of context for
> >>>>> the open() callback
> >>>>>
> >>>>> What about this:
> >>>>> Open the dev_ctx in the tee_device_register(), at the last step before
> >>>>> setting the TEE_DEVICE_FLAG_REGISTERED flag. Then the open() callback can
> >>>>> check for this flag to determine if it is a normal context or dev_ctx.
> >>>>> If the open() is called while the device has not been registered, it should
> >>>>> handle it differently
> >>>>
> >>>> That makes sense, the driver should be prepared to handle open() calls
> >>>> after tee_device_register() anyway.
> >>>> However, there is no serialization of the flags field in struct
> >>>> tee_device. Hmm, would it be too hacky for the open() callback to
> >>>> check if &ctx->teedev.dev_ctx == ctx? We could add a helper function
> >>>> to wrap that check.
> >>>>
> >>>
> >>> Your suggested change requires every driver to update open() callback
> >>> and later other callbacks may have to support it too. IMHO, only
> >>> teedev_get_dev_ctx() should be able to return a reference to device
> >>> context for usage within the TEE and the implementation driver.
> >>
> >> Yes, but it's only the OP-TEE driver that needs anything special. It
> >> looks like the others can be left unchanged.
> >
> > I suppose it's most likely the upcoming QTEE driver requiring it.
> >
>
> I don't believe this is correct. This requirement is implicitly imposed
> by the TEE subsystem API. If calling open() is acceptable in
> tee_device_alloc(), then I could argue that tee_device_register() and
> tee_device_alloc() should be merged into a single function. If a driver
> is ready to handle requests, why delay its exposure by postponing the
> registration?

You can't mix in device alloc/init with device registration. As soon
as you register a device, it's available for user-space to issue
IOCTLs. So if there is any race between device init and it's usage
then it's going to cause hard to debug issues.

There can be an argument that dev_ctx being a member of "struct
tee_device" so it should get initialized alongside other bits in
tee_device_alloc() (we can rename
s/tee_device_alloc()/tee_device_init()/ if that makes it obvious). But
I can live with the device context getting initialized as the first
thing in tee_device_register() avoiding the race window mentioned
above.

>
> By calling open() in tee_device_alloc(), you indirectly impose an unspoken
> requirement on developers regarding how they should write their drivers,
> such as the steps they should take to probe the device.

Still not sure why it isn't an explicit requirement when you are
already passing an initialized ops structure to tee_device_alloc().

-Sumit

>
> Regards,
> Amir
>
> >>
> >>>
> >>> I am still not able to understand why the following won't work with a
> >>> clear lifetime for the device context?
> >>>
> >>> tee_device_alloc()
> >>>   -> if (!(teedesc->flags & TEE_DESC_PRIVILEGED))
> >>> desc->ops->open(&teedev->dev_ctx);
> >>
> >> We must also have a fully initialized dev_ctx for the supplicant
> >> device.
> >
> > Currently I only see following for OP-TEE driver:
> >
> > ctx = teedev_open(optee->teedev);
> >
> > And I can't see anything like below:
> >
> > ctx = teedev_open(optee->supp_teedev);
> >
> > Where do you think that the dev_ctx is required for a supplicant
> > device? AFAICS, currently opening a context with the supplicant device
> > means that the supplicant daemon is available to handle RPCs which
> > won't be possible during OP-TEE driver probe. Am I missing something?
> >
> >> I'd rather delay the open() callback until
> >> tee_device_register() since the dev_ctx is guaranteed not to be needed
> >> before that.
> >
> > Okay, the updated call chain can look like:
> >
> > tee_device_register()
> >   -> if (!(teedev->desc->flags & TEE_DESC_PRIVILEGED))
> >          desc->ops->open(&teedev->dev_ctx);
> >>
> >>>
> >>> tee_device_put()
> >>>   -> if (teedev->dev_ctx) desc->ops->release(&teedev->dev_ctx);
> >>
> >> teedev->dev_ctx is supposed to be embedded in struct tee_device, so
> >> the if isn't needed.
> >
> > I added "if" to cover the case when dev_ctx is not initialized for the
> > supplicant device.
> >
> > -Sumit
> >
> >>
> >> Cheers,
> >> Jens
> >>
> >>>
> >>> -Sumit
> >>>
> >>>> Cheers,
> >>>> Jens
> >>>>
> >>>>>
> >>>>> - Amir
> >>>>>
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> Jens
> >>>>>>>
> >>>>>>>>
> >>>>>>>> -Sumit

Re: [PATCH RFC 3/3] tee: introduce orphan tee_shm and default context

[Amirreza Zarrabi]

On 11/28/2024 11:44 PM, Sumit Garg wrote:
> On Thu, 28 Nov 2024 at 02:29, Amirreza Zarrabi
> <quic_azarrabi@quicinc.com> wrote:
>>
>>
>>
>> On 11/27/2024 5:01 PM, Sumit Garg wrote:
>>> On Tue, 26 Nov 2024 at 20:52, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>>>
>>>> On Tue, Nov 26, 2024 at 1:27 PM Sumit Garg <sumit.garg@linaro.org> wrote:
>>>>>
>>>>> On Tue, 26 Nov 2024 at 14:03, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>>>>>
>>>>>> On Mon, Nov 25, 2024 at 9:55 PM Amirreza Zarrabi
>>>>>> <quic_azarrabi@quicinc.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 11/25/2024 6:51 PM, Sumit Garg wrote:
>>>>>>>> On Mon, 25 Nov 2024 at 12:53, Jens Wiklander <jens.wiklander@linaro.org> wrote:
>>>>>>>>>
>>>>>>>>> On Mon, Nov 25, 2024 at 7:14 AM Sumit Garg <sumit.garg@linaro.org> wrote:
>>>>>>>>>>
>>>>>>>>>> On Mon, 25 Nov 2024 at 03:00, Amirreza Zarrabi
>>>>>>>>>> <quic_azarrabi@quicinc.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi Sumit,
>>>>>>>>>>>
>>>>>>>>>>> Thank you so much for the comemnts :).
>>>>>>>>>>>
>>>>>>>>>>> On 11/23/2024 9:32 PM, Sumit Garg wrote:
>>>>>>>>>>>> Hi Amirreza,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for proposing this.
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, 22 Nov 2024 at 06:38, Amirreza Zarrabi
>>>>>>>>>>>> <quic_azarrabi@quicinc.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 11/21/2024 11:08 PM, Jens Wiklander wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Jens,
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Amirreza,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Nov 21, 2024 at 2:37 AM Amirreza Zarrabi
>>>>>>>>>>>>>> <quic_azarrabi@quicinc.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The default context has a lifespan similar to the tee_device.
>>>>>>>>>>>>
>>>>>>>>>>>> Since it's associated with tee_device context, let's call it obvious
>>>>>>>>>>>> via renaming it as device context instead (s/def_ctx/dev_ctx/ in this
>>>>>>>>>>>> patch).
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Make sense, I'll rename it.
>>>>>>>>>>>
>>>>>>>>>>>>>>> It is used as a context for shared memory if the context to which the
>>>>>>>>>>>>>>> shared memory belongs is released, making the tee_shm an orphan.
>>>>>>>>>>>>>>> This allows the driver implementing shm_unregister to safely make
>>>>>>>>>>>>>>> subsequent calls, such as to a supplicant if needed.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It also enables users to free the shared memory while the driver is
>>>>>>>>>>>>>>> blocked on unregister_tee_device safely.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Preferably, this should be used for all driver internal uses, using
>>>>>>>>>>>>>>> teedev_get_def_context rather than calling teedev_open.
>>>>>>>>>>>>
>>>>>>>>>>>> Makes sense to me.
>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Signed-off-by: Amirreza Zarrabi <quic_azarrabi@quicinc.com>
>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>  drivers/tee/optee/core.c    |  2 +-
>>>>>>>>>>>>>>>  drivers/tee/optee/ffa_abi.c |  2 +-
>>>>>>>>>>>>>>>  drivers/tee/optee/smc_abi.c |  2 +-
>>>>>>>>>>>>>>>  drivers/tee/tee_core.c      | 83 +++++++++++++++++++++++++++++----------------
>>>>>>>>>>>>>>>  drivers/tee/tee_private.h   |  3 --
>>>>>>>>>>>>>>>  drivers/tee/tee_shm.c       | 18 ++--------
>>>>>>>>>>>>>>>  include/linux/tee_core.h    | 15 ++++++++
>>>>>>>>>>>>>>>  include/linux/tee_drv.h     |  7 ----
>>>>>>>>>>>>>>>  8 files changed, 73 insertions(+), 59 deletions(-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> diff --git a/drivers/tee/optee/core.c b/drivers/tee/optee/core.c
>>>>>>>>>>>>>>> index c75fddc83576..78d43d0c8014 100644
>>>>>>>>>>>>>>> --- a/drivers/tee/optee/core.c
>>>>>>>>>>>>>>> +++ b/drivers/tee/optee/core.c
>>>>>>>>>>>>>>> @@ -173,7 +173,7 @@ void optee_remove_common(struct optee *optee)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>         optee_notif_uninit(optee);
>>>>>>>>>>>>>>>         optee_shm_arg_cache_uninit(optee);
>>>>>>>>>>>>>>> -       teedev_close_context(optee->ctx);
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>         /*
>>>>>>>>>>>>>>>          * The two devices have to be unregistered before we can free the
>>>>>>>>>>>>>>>          * other resources.
>>>>>>>>>>>>>>> diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
>>>>>>>>>>>>>>> index f3af5666bb11..6ad94f0788ad 100644
>>>>>>>>>>>>>>> --- a/drivers/tee/optee/ffa_abi.c
>>>>>>>>>>>>>>> +++ b/drivers/tee/optee/ffa_abi.c
>>>>>>>>>>>>>>> @@ -949,7 +949,7 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
>>>>>>>>>>>>>>>         optee_shm_arg_cache_init(optee, arg_cache_flags);
>>>>>>>>>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
>>>>>>>>>>>>>>>         ffa_dev_set_drvdata(ffa_dev, optee);
>>>>>>>>>>>>>>> -       ctx = teedev_open(optee->teedev);
>>>>>>>>>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
>>>>>>>>>>>>>>>         if (IS_ERR(ctx)) {
>>>>>>>>>>>>>>>                 rc = PTR_ERR(ctx);
>>>>>>>>>>>>>>>                 goto err_rhashtable_free;
>>>>>>>>>>>>>>> diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
>>>>>>>>>>>>>>> index e9456e3e74cc..c77a3e631d04 100644
>>>>>>>>>>>>>>> --- a/drivers/tee/optee/smc_abi.c
>>>>>>>>>>>>>>> +++ b/drivers/tee/optee/smc_abi.c
>>>>>>>>>>>>>>> @@ -1722,7 +1722,7 @@ static int optee_probe(struct platform_device *pdev)
>>>>>>>>>>>>>>>         mutex_init(&optee->rpmb_dev_mutex);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>         platform_set_drvdata(pdev, optee);
>>>>>>>>>>>>>>> -       ctx = teedev_open(optee->teedev);
>>>>>>>>>>>>>>> +       ctx = teedev_get_def_context(optee->teedev);
>>>>>>>>>>>>>>>         if (IS_ERR(ctx)) {
>>>>>>>>>>>>>>>                 rc = PTR_ERR(ctx);
>>>>>>>>>>>>>>>                 goto err_supp_uninit;
>>>>>>>>>>>>>>> diff --git a/drivers/tee/tee_core.c b/drivers/tee/tee_core.c
>>>>>>>>>>>>>>> index 93f3b330aec8..805e1336089d 100644
>>>>>>>>>>>>>>> --- a/drivers/tee/tee_core.c
>>>>>>>>>>>>>>> +++ b/drivers/tee/tee_core.c
>>>>>>>>>>>>>>> @@ -57,7 +57,6 @@ struct tee_context *teedev_open(struct tee_device *teedev)
>>>>>>>>>>>>>>>                 goto err;
>>>>>>>>>>>>>>>         }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -       kref_init(&ctx->refcount);
>>>>>>>>>>>>>>>         ctx->teedev = teedev;
>>>>>>>>>>>>>>>         INIT_LIST_HEAD(&ctx->list_shm);
>>>>>>>>>>>>>>>         rc = teedev->desc->ops->open(ctx);
>>>>>>>>>>>>>>> @@ -73,36 +72,43 @@ struct tee_context *teedev_open(struct tee_device *teedev)
>>>>>>>>>>>>>>>  }
>>>>>>>>>>>>>>>  EXPORT_SYMBOL_GPL(teedev_open);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -void teedev_ctx_get(struct tee_context *ctx)
>>>>>>>>>>>>>>> +struct tee_context *teedev_get_def_context(struct tee_device *teedev)
>>>>>>>>>>>>>>>  {
>>>>>>>>>>>>>>> -       if (ctx->releasing)
>>>>>>>>>>>>>>> -               return;
>>>>>>>>>>>>>>> +       int rc;
>>>>>>>>>>>>>>> +       struct tee_context *ctx = &teedev->def_ctx;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -       kref_get(&ctx->refcount);
>>>>>>>>>>>>>>> -}
>>>>>>>>>>>>>>> +       ctx->teedev = teedev;
>>>>>>>>>>>>>>> +       INIT_LIST_HEAD(&ctx->list_shm);
>>>>>>>>>>>>>>> +       rc = teedev->desc->ops->open(ctx);
>>>>>>>>>>>>>>> +       if (rc)
>>>>>>>>>>>>>>> +               return ERR_PTR(rc);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think ctx->teedev and ctx->list_shm must always be initialized or
>>>>>>>>>>>>>> &teedev->def_ctx can't be used in teedev_close_context().
>>>>>>>>>>>>>
>>>>>>>>>>>>> True, but &teedev->def_ctx is never used in teedev_close_context().
>>>>>>>>>>>>> The closing of the &teedev->def_ctx simply ignored. So once opened,
>>>>>>>>>>>>> &teedev->def_ctx will always remain open until the tee_device is alive.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> We could initialize teedev->def_ctx on the first call to teedev_open()
>>>>>>>>>>>>>> on that tee_device. We need a way to tell the
>>>>>>>>>>>>>> teedev->desc->ops->open() to the backed driver that it's initializing
>>>>>>>>>>>>>> the default context though, or optee_open() can't handle the
>>>>>>>>>>>>>> tee-supplicant case properly.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> That's a good point. This way, it is guaranteed that there is one def_ctx
>>>>>>>>>>>>> per teedev. There should be a way to tell the open() callback that it is
>>>>>>>>>>>>> a def_ctx, so it is not registered as a supplicant context.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Should we allow this function to be called more than once for each teedev?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, moving to teedev_open() will fix the issue.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Do we need serialization in this function if it's called after the
>>>>>>>>>>>>>> driver is probed?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> True. I'll make sure there is no race.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -static void teedev_ctx_release(struct kref *ref)
>>>>>>>>>>>>>>> -{
>>>>>>>>>>>>>>> -       struct tee_context *ctx = container_of(ref, struct tee_context,
>>>>>>>>>>>>>>> -                                              refcount);
>>>>>>>>>>>>>>> -       ctx->releasing = true;
>>>>>>>>>>>>>>> -       ctx->teedev->desc->ops->release(ctx);
>>>>>>>>>>>>>>> -       kfree(ctx);
>>>>>>>>>>>>>>> +       return ctx;
>>>>>>>>>>>>>>>  }
>>>>>>>>>>>>>>> +EXPORT_SYMBOL_GPL(teedev_get_def_context);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -void teedev_ctx_put(struct tee_context *ctx)
>>>>>>>>>>>>>>> +void teedev_close_context(struct tee_context *ctx)
>>>>>>>>>>>>>>>  {
>>>>>>>>>>>>>>> -       if (ctx->releasing)
>>>>>>>>>>>>>>> +       struct tee_device *teedev = ctx->teedev;
>>>>>>>>>>>>>>> +       struct tee_shm *shm;
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>> +       if (ctx == &teedev->def_ctx)
>>>>>>>>>>>>>>>                 return;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -       kref_put(&ctx->refcount, teedev_ctx_release);
>>>>>>>>>>>>>>> -}
>>>>>>>>>>>>>>> +       teedev->desc->ops->release(ctx);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -void teedev_close_context(struct tee_context *ctx)
>>>>>>>>>>>>>>> -{
>>>>>>>>>>>>>>> -       struct tee_device *teedev = ctx->teedev;
>>>>>>>>>>>>>>> +       mutex_lock(&teedev->mutex);
>>>>>>>>>>>>>>> +       list_for_each_entry(shm, &ctx->list_shm, link) {
>>>>>>>>>>>>>>> +               /* Context released. However, shm still holding a teedev reference.
>>>>>>>>>>>>>>> +                * Replace shm->ctx with the default context so that tee_shm_get_from_id()
>>>>>>>>>>>>>>> +                * fails (i.e. it is not accessible from userspace) but shm still
>>>>>>>>>>>>>>> +                * holds a valid context for further clean up, e.g. shm_unregister().
>>>>>>>>>>>>>>> +                */
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> /*
>>>>>>>>>>>>>>  * Please format
>>>>>>>>>>>>>>  * multiline comments
>>>>>>>>>>>>>>  * like this. Please
>>>>>>>>>>>>>>  * keep the lines at
>>>>>>>>>>>>>>  * max 80 columns
>>>>>>>>>>>>>>  * here and at other
>>>>>>>>>>>>>>  * places in the patch-
>>>>>>>>>>>>>>  * set.
>>>>>>>>>>>>>>  */
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ack.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> +               shm->ctx = &teedev->def_ctx;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> shm->ctx will always point to a valid context, even if it is the
>>>>>>>>>>>>>> default context. It seems that we can always get hold of the correct
>>>>>>>>>>>>>> teedev via shm->ctx->teedev. Do we need "tee: revert removal of
>>>>>>>>>>>>>> redundant teedev in struct tee_shm"?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> It was there in case we wanted to use NULL, but with def_ctx, it is not
>>>>>>>>>>>>> necessary. I am withdrawing that commit. :).
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Shouldn't the shm be removed from the ctx->list_shm and be moved to
>>>>>>>>>>>>>> teedev->def_ctx.list_shm?
>>>>>>>>>>>>
>>>>>>>>>>>> +1
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Ack.
>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Not really. If we put shm in the teedev->def_ctx.list_shm, by the time
>>>>>>>>>>>>> we are closing the def_ctx, the list is guaranteed to be empty.
>>>>>>>>>>>>>
>>>>>>>>>>>>> However, I understand it is cleaner and more consistent to do that rather
>>>>>>>>>>>>> than making changes to tee_shm_put().
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'll do it.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> +       }
>>>>>>>>>>>>>>> +       mutex_unlock(&teedev->mutex);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -       teedev_ctx_put(ctx);
>>>>>>>>>>>>>>> +       kfree(ctx);
>>>>>>>>>>>>>>>         tee_device_put(teedev);
>>>>>>>>>>>>>>>  }
>>>>>>>>>>>>>>>  EXPORT_SYMBOL_GPL(teedev_close_context);
>>>>>>>>>>>>>>> @@ -946,6 +952,8 @@ struct tee_device *tee_device_alloc(const struct tee_desc *teedesc,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>         teedev->desc = teedesc;
>>>>>>>>>>>>>>>         teedev->pool = pool;
>>>>>>>>>>>>>>> +       /* Only open default context when teedev_get_def_context() called. */
>>>>>>>>>>>>>>> +       teedev->def_ctx.teedev = NULL;
>>>>>>>>>>>>
>>>>>>>>>>>> Why don't you open the device context here only? This will associate
>>>>>>>>>>>> it automatically with teedev lifespan and then
>>>>>>>>>>>> teedev_get_def_context() will just return a reference to that.
>>>>>>>>>>>>
>>>>>>>>>>>> -Sumit
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> So my assumption is that the tee_devic_alloc() is called as part of
>>>>>>>>>>> the driver initialization; there is no guarantee that at this time the
>>>>>>>>>>> driver is actually ready to accept any open() callback.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The drivers should be able to handle open() callback since we already
>>>>>>>>>> check for !teedesc->ops->open in the beginning of tee_devic_alloc().
>>>>>>>>>> Also, we need to open a device context for !TEE_DESC_PRIVILEGED such
>>>>>>>>>> that we don't open a supplicant device context there.
>>>>>>>>>
>>>>>>>>> It would be nice to have the device context fully initialized when the
>>>>>>>>> probe function returns. How about adding a "bool is_dev_ctx" to struct
>>>>>>>>> tee_context so the open() callback can tell that this is a special
>>>>>>>>> tee_contex?
>>>>>>>>
>>>>>>>> Sure, that will be useful to distinguish the device context from
>>>>>>>> normal client context.
>>>>>>>>
>>>>>>>> -Sumit
>>>>>>>>
>>>>>>>
>>>>>>> So, as far as the open() callback, I do not believe checking if it is not null
>>>>>>> is reasonable for calling it here. Most drivers allocate resources and then
>>>>>>> initialize them. So, assume these steps for a TEE driver:
>>>>>>>  (1) allocate internal data structures,
>>>>>>>  (2) allocate the device,
>>>>>>>  (3) initialize the internal data structurse and then
>>>>>>>  (4) register the device.
>>>>>>>
>>>>>>> Having these steps for a backend driver means that if you call open() at
>>>>>>> step (2), the internal data structures are not ready.
>>>>>
>>>>> As part of tee_device_alloc(), every driver has to pass "const struct
>>>>> tee_desc *teedesc" fully initialized. Which internal data structures
>>>>> are you referring too? Is there any upstream example?
>>>>
>>>> It's reasonable to wait with the open() callback until step 4 above,
>>>> which should correspond with the tee_device_register() call. Data
>>>> written only once doesn't need serialized access if the fields are
>>>> only accessed after they have been fully initialized.
>>>
>>> Fair enough, I can live with the device context opened after registering it.
>>>
>>>>
>>>>>
>>>>>>>
>>>>>>> I was originally thinking of going with Jens' suggestion to open dev_ctx in
>>>>>>> the teedev_open(), and use a flag to distinguish the type of context for
>>>>>>> the open() callback
>>>>>>>
>>>>>>> What about this:
>>>>>>> Open the dev_ctx in the tee_device_register(), at the last step before
>>>>>>> setting the TEE_DEVICE_FLAG_REGISTERED flag. Then the open() callback can
>>>>>>> check for this flag to determine if it is a normal context or dev_ctx.
>>>>>>> If the open() is called while the device has not been registered, it should
>>>>>>> handle it differently
>>>>>>
>>>>>> That makes sense, the driver should be prepared to handle open() calls
>>>>>> after tee_device_register() anyway.
>>>>>> However, there is no serialization of the flags field in struct
>>>>>> tee_device. Hmm, would it be too hacky for the open() callback to
>>>>>> check if &ctx->teedev.dev_ctx == ctx? We could add a helper function
>>>>>> to wrap that check.
>>>>>>
>>>>>
>>>>> Your suggested change requires every driver to update open() callback
>>>>> and later other callbacks may have to support it too. IMHO, only
>>>>> teedev_get_dev_ctx() should be able to return a reference to device
>>>>> context for usage within the TEE and the implementation driver.
>>>>
>>>> Yes, but it's only the OP-TEE driver that needs anything special. It
>>>> looks like the others can be left unchanged.
>>>
>>> I suppose it's most likely the upcoming QTEE driver requiring it.
>>>
>>
>> I don't believe this is correct. This requirement is implicitly imposed
>> by the TEE subsystem API. If calling open() is acceptable in
>> tee_device_alloc(), then I could argue that tee_device_register() and
>> tee_device_alloc() should be merged into a single function. If a driver
>> is ready to handle requests, why delay its exposure by postponing the
>> registration?
> 
> You can't mix in device alloc/init with device registration. As soon
> as you register a device, it's available for user-space to issue
> IOCTLs. So if there is any race between device init and it's usage
> then it's going to cause hard to debug issues.
> 

That's exactly my point :). Whoever alloc the device, knows that the
device is not being available untill the regsiteration is done. So they
may make decisions based on that, e.g reorder some init steps.

> There can be an argument that dev_ctx being a member of "struct
> tee_device" so it should get initialized alongside other bits in
> tee_device_alloc() (we can rename
> s/tee_device_alloc()/tee_device_init()/ if that makes it obvious). But
> I can live with the device context getting initialized as the first
> thing in tee_device_register() avoiding the race window mentioned
> above.
> 
>>
>> By calling open() in tee_device_alloc(), you indirectly impose an unspoken
>> requirement on developers regarding how they should write their drivers,
>> such as the steps they should take to probe the device.
> 
> Still not sure why it isn't an explicit requirement when you are
> already passing an initialized ops structure to tee_device_alloc().
> 
> -Sumit

Whoeever, passed the ops to the alloc interface may assume that it is
not being called untill the register being finished as you mentioned
above. I assume it is fair assumption.

Regards,
Amir

> 
>>
>> Regards,
>> Amir
>>
>>>>
>>>>>
>>>>> I am still not able to understand why the following won't work with a
>>>>> clear lifetime for the device context?
>>>>>
>>>>> tee_device_alloc()
>>>>>   -> if (!(teedesc->flags & TEE_DESC_PRIVILEGED))
>>>>> desc->ops->open(&teedev->dev_ctx);
>>>>
>>>> We must also have a fully initialized dev_ctx for the supplicant
>>>> device.
>>>
>>> Currently I only see following for OP-TEE driver:
>>>
>>> ctx = teedev_open(optee->teedev);
>>>
>>> And I can't see anything like below:
>>>
>>> ctx = teedev_open(optee->supp_teedev);
>>>
>>> Where do you think that the dev_ctx is required for a supplicant
>>> device? AFAICS, currently opening a context with the supplicant device
>>> means that the supplicant daemon is available to handle RPCs which
>>> won't be possible during OP-TEE driver probe. Am I missing something?
>>>
>>>> I'd rather delay the open() callback until
>>>> tee_device_register() since the dev_ctx is guaranteed not to be needed
>>>> before that.
>>>
>>> Okay, the updated call chain can look like:
>>>
>>> tee_device_register()
>>>   -> if (!(teedev->desc->flags & TEE_DESC_PRIVILEGED))
>>>          desc->ops->open(&teedev->dev_ctx);
>>>>
>>>>>
>>>>> tee_device_put()
>>>>>   -> if (teedev->dev_ctx) desc->ops->release(&teedev->dev_ctx);
>>>>
>>>> teedev->dev_ctx is supposed to be embedded in struct tee_device, so
>>>> the if isn't needed.
>>>
>>> I added "if" to cover the case when dev_ctx is not initialized for the
>>> supplicant device.
>>>
>>> -Sumit
>>>
>>>>
>>>> Cheers,
>>>> Jens
>>>>
>>>>>
>>>>> -Sumit
>>>>>
>>>>>> Cheers,
>>>>>> Jens
>>>>>>
>>>>>>>
>>>>>>> - Amir
>>>>>>>
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Jens
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -Sumit