[PATCH 06/20] media: atomisp: Fix deadlock when the /dev/video# node is closed while still streaming

[Hans de Goede <hdegoede@redhat.com>]

atomisp_release() was taking pipe->vb_queue_mutex + isp->mutex at the
same time. But if the /dev/video# node is closed while still streaming
then vb2_queue_release() will call atomisp_stop_streaming() which takes
isp->mutex itself, leading to a deadlock.

To fix this only take isp->mutex after cleaning up the v4l2_fh /
the vb2_queue. While at it switch to vb2_fop_release() which will take
pipe->vb_queue_mutex for us, which also resolves a FIXME comment.

Signed-off-by: Hans de Goede <hdegoede@redhat.com>
---
 .../staging/media/atomisp/pci/atomisp_fops.c  | 21 +++++--------------
 1 file changed, 5 insertions(+), 16 deletions(-)

diff --git a/drivers/staging/media/atomisp/pci/atomisp_fops.c b/drivers/staging/media/atomisp/pci/atomisp_fops.c
index 21ca276831ce..8cff26d42b82 100644
--- a/drivers/staging/media/atomisp/pci/atomisp_fops.c
+++ b/drivers/staging/media/atomisp/pci/atomisp_fops.c
@@ -861,23 +861,14 @@ static int atomisp_release(struct file *file)
 
 	v4l2_fh_init(&fh.vfh, vdev);
 
-	mutex_lock(&pipe->vb_queue_mutex);
-	mutex_lock(&isp->mutex);
-
 	dev_dbg(isp->dev, "release device %s\n", vdev->name);
 
 	asd->subdev.devnode = vdev;
 
-	/*
-	 * FIXME This if is copied from _vb2_fop_release, this cannot use that
-	 * because that calls v4l2_fh_release() earlier then this function.
-	 * Maybe we can release the fh earlier though, it does not look like
-	 * anything needs it after this.
-	 */
-	if (file->private_data == vdev->queue->owner) {
-		vb2_queue_release(vdev->queue);
-		vdev->queue->owner = NULL;
-	}
+	/* Note file must not be used after this! */
+	vb2_fop_release(file);
+
+	mutex_lock(&isp->mutex);
 
 	pipe->users--;
 	if (pipe->users)
@@ -940,9 +931,7 @@ static int atomisp_release(struct file *file)
 				     V4L2_SEL_TGT_COMPOSE, 0,
 				     &clear_compose);
 	mutex_unlock(&isp->mutex);
-	mutex_unlock(&pipe->vb_queue_mutex);
-
-	return v4l2_fh_release(file);
+	return 0;
 }
 
 const struct v4l2_file_operations atomisp_fops = {
-- 
2.38.1