[PATCH 0/4] Drivers: hv: utils: prevent crash when a utility driver is disabled host side

[PATCH 0/4] Drivers: hv: utils: prevent crash when a utility driver is disabled host side

[Vitaly Kuznetsov]

I'm observing a crash when a utility driver is disabled host side (e.g.
'Guest services' is disabled live) when we have userspace daemon
connected:

[   90.244859] general protection fault: 0000 [#1] SMP
...
[   90.800082] CPU: 3 PID: 473 Comm: hypervfcopyd Not tainted 4.3.0-rc7_netvsc_noalloc+ #1053
...
[   90.800082] Call Trace:
[   90.800082]  [<ffffffff81187008>] __fput+0xc8/0x1f0
[   90.800082]  [<ffffffff8118716e>] ____fput+0xe/0x10
[   90.800082]  [<ffffffff8107e4b3>] task_work_run+0x73/0x90
[   90.800082]  [<ffffffff81066995>] do_exit+0x335/0xa90
[   90.800082]  [<ffffffff8106716f>] do_group_exit+0x3f/0xc0
[   90.800082]  [<ffffffff81071abe>] get_signal+0x25e/0x5e0
[   90.800082]  [<ffffffff81015278>] do_signal+0x28/0x580
[   90.800082]  [<ffffffff81086656>] ? finish_task_switch+0xa6/0x180
[   90.800082]  [<ffffffff81443ebf>] ? __schedule+0x28f/0x870
[   90.800082]  [<ffffffffa01ebbaa>] ? hvt_op_read+0x12a/0x140 [hv_utils]
[   90.800082]  [<ffffffff8109eca0>] ? wake_atomic_t_function+0x70/0x70
...
[   90.800082] RIP  [<ffffffff810d3866>] module_put+0x16/0x90
[   90.800082]  RSP <ffff88003eb1bb88>
[   95.734261] ---[ end trace 0e09af6a6294a668 ]---

The problem is that hvutil_transport_destroy() which does misc_deregister()
freeing the appropriate device is reachable by two paths: module unload
and from util_remove(). While module unload path is protected by .owner in
struct file_operations util_remove() path is not. Freeing the device while
someone holds an open fd for it is a show stopper.

In general, it is not possible to revoke an fd from all users so the only
way to solve the issue is to defer freeing the hvutil_transport structure
asking the daemon to exit gracefully by responding -EBADF to all
operations on unload.

Patch 1 fixes an unrelated issue I spotted, patch 2 renames outmsg_lock to
'lock' as we're gonna use it to protect test-and-set operations on 'mode',
patch 3 introduces HVUTIL_TRANSPORT_DESTROY mode of operation, patch 4
fixes the issue itself.

Patches are rebased on previously sent Olaf's fixes.

Vitaly Kuznetsov (4):
  Drivers: hv: utils: fix memory leak on on_msg() failure
  Drivers: hv: utils: rename outmsg_lock
  Drivers: hv: utils: introduce HVUTIL_TRANSPORT_DESTROY mode
  Drivers: hv: utils: fix crash when device is removed from host side

 drivers/hv/hv_utils_transport.c | 110 +++++++++++++++++++++++++++++++---------
 drivers/hv/hv_utils_transport.h |   3 +-
 2 files changed, 88 insertions(+), 25 deletions(-)

-- 
2.4.3

[PATCH 1/4] Drivers: hv: utils: fix memory leak on on_msg() failure

[Vitaly Kuznetsov]

inmsg should be freed in case of on_msg() failure to avoid memory leak.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 drivers/hv/hv_utils_transport.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/hv/hv_utils_transport.c b/drivers/hv/hv_utils_transport.c
index 24b2766..1d20451 100644
--- a/drivers/hv/hv_utils_transport.c
+++ b/drivers/hv/hv_utils_transport.c
@@ -77,6 +77,7 @@ static ssize_t hvt_op_write(struct file *file, const char __user *buf,
 {
 	struct hvutil_transport *hvt;
 	u8 *inmsg;
+	ssize_t ret = count;
 
 	hvt = container_of(file->f_op, struct hvutil_transport, fops);
 
@@ -85,10 +86,10 @@ static ssize_t hvt_op_write(struct file *file, const char __user *buf,
 		return PTR_ERR(inmsg);
 
 	if (hvt->on_msg(inmsg, count))
-		return -EFAULT;
+		ret = -EFAULT;
 	kfree(inmsg);
 
-	return count;
+	return ret;
 }
 
 static unsigned int hvt_op_poll(struct file *file, poll_table *wait)
-- 
2.4.3

Re: [PATCH 1/4] Drivers: hv: utils: fix memory leak on on_msg() failure

[Dan Carpenter]

On Thu, Nov 12, 2015 at 12:32:13PM +0100, Vitaly Kuznetsov wrote:
> @@ -85,10 +86,10 @@ static ssize_t hvt_op_write(struct file *file, const char __user *buf,
>  		return PTR_ERR(inmsg);
>  
>  	if (hvt->on_msg(inmsg, count))
> -		return -EFAULT;
> +		ret = -EFAULT;

You fix this leak and then re-introduce it again directly in patch 3/4.
Also it might be nice to preserve the error code.

	ret = hvt->on_msg(inmsg, count);

	kfree(inmsg);

	return ret ? ret : count;

regards,
dan carpenter

Re: [PATCH 1/4] Drivers: hv: utils: fix memory leak on on_msg() failure

[Vitaly Kuznetsov]

Dan Carpenter <dan.carpenter@oracle.com> writes:

> On Thu, Nov 12, 2015 at 12:32:13PM +0100, Vitaly Kuznetsov wrote:
>> @@ -85,10 +86,10 @@ static ssize_t hvt_op_write(struct file *file, const char __user *buf,
>>  		return PTR_ERR(inmsg);
>>  
>>  	if (hvt->on_msg(inmsg, count))
>> -		return -EFAULT;
>> +		ret = -EFAULT;
>
> You fix this leak and then re-introduce it again directly in patch
> 3/4.

Such a shame ... Thanks for noticing!

> Also it might be nice to preserve the error code.
>
> 	ret = hvt->on_msg(inmsg, count);
>
> 	kfree(inmsg);
>
> 	return ret ? ret : count;

It seems on_msg() hooks can only return -EINVAL now but why not
... let's do it in v2.

-- 
  Vitaly

[PATCH 2/4] Drivers: hv: utils: rename outmsg_lock

[Vitaly Kuznetsov]

As a preparation to reusing outmsg_lock to protect test-and-set openrations
on 'mode' rename it the more general 'lock'.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 drivers/hv/hv_utils_transport.c | 14 +++++++-------
 drivers/hv/hv_utils_transport.h |  2 +-
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/drivers/hv/hv_utils_transport.c b/drivers/hv/hv_utils_transport.c
index 1d20451..5cce2d2 100644
--- a/drivers/hv/hv_utils_transport.c
+++ b/drivers/hv/hv_utils_transport.c
@@ -27,11 +27,11 @@ static struct list_head hvt_list = LIST_HEAD_INIT(hvt_list);
 
 static void hvt_reset(struct hvutil_transport *hvt)
 {
-	mutex_lock(&hvt->outmsg_lock);
+	mutex_lock(&hvt->lock);
 	kfree(hvt->outmsg);
 	hvt->outmsg = NULL;
 	hvt->outmsg_len = 0;
-	mutex_unlock(&hvt->outmsg_lock);
+	mutex_unlock(&hvt->lock);
 	if (hvt->on_reset)
 		hvt->on_reset();
 }
@@ -47,7 +47,7 @@ static ssize_t hvt_op_read(struct file *file, char __user *buf,
 	if (wait_event_interruptible(hvt->outmsg_q, hvt->outmsg_len > 0))
 		return -EINTR;
 
-	mutex_lock(&hvt->outmsg_lock);
+	mutex_lock(&hvt->lock);
 	if (!hvt->outmsg) {
 		ret = -EAGAIN;
 		goto out_unlock;
@@ -68,7 +68,7 @@ static ssize_t hvt_op_read(struct file *file, char __user *buf,
 	hvt->outmsg_len = 0;
 
 out_unlock:
-	mutex_unlock(&hvt->outmsg_lock);
+	mutex_unlock(&hvt->lock);
 	return ret;
 }
 
@@ -197,7 +197,7 @@ int hvutil_transport_send(struct hvutil_transport *hvt, void *msg, int len)
 		return ret;
 	}
 	/* HVUTIL_TRANSPORT_CHARDEV */
-	mutex_lock(&hvt->outmsg_lock);
+	mutex_lock(&hvt->lock);
 	if (hvt->outmsg) {
 		/* Previous message wasn't received */
 		ret = -EFAULT;
@@ -211,7 +211,7 @@ int hvutil_transport_send(struct hvutil_transport *hvt, void *msg, int len)
 	} else
 		ret = -ENOMEM;
 out_unlock:
-	mutex_unlock(&hvt->outmsg_lock);
+	mutex_unlock(&hvt->lock);
 	return ret;
 }
 
@@ -242,7 +242,7 @@ struct hvutil_transport *hvutil_transport_init(const char *name,
 	hvt->mdev.fops = &hvt->fops;
 
 	init_waitqueue_head(&hvt->outmsg_q);
-	mutex_init(&hvt->outmsg_lock);
+	mutex_init(&hvt->lock);
 
 	spin_lock(&hvt_list_lock);
 	list_add(&hvt->list, &hvt_list);
diff --git a/drivers/hv/hv_utils_transport.h b/drivers/hv/hv_utils_transport.h
index 314c76c..bff4c92 100644
--- a/drivers/hv/hv_utils_transport.h
+++ b/drivers/hv/hv_utils_transport.h
@@ -38,7 +38,7 @@ struct hvutil_transport {
 	u8 *outmsg;                         /* message to the userspace */
 	int outmsg_len;                     /* its length */
 	wait_queue_head_t outmsg_q;         /* poll/read wait queue */
-	struct mutex outmsg_lock;           /* protects outmsg */
+	struct mutex lock;                  /* protects struct members */
 };
 
 struct hvutil_transport *hvutil_transport_init(const char *name,
-- 
2.4.3

[PATCH 3/4] Drivers: hv: utils: introduce HVUTIL_TRANSPORT_DESTROY mode

[Vitaly Kuznetsov]

When Hyper-V host asks us to remove some util driver by closing the
appropriate channel there is no easy way to force the current file
descriptor holder to hang up but we can start to respond -EBADF to all
operations asking it to exit gracefully.

As we're setting hvt->mode from two separate contexts now we need to use
a proper locking.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 drivers/hv/hv_utils_transport.c | 69 ++++++++++++++++++++++++++++++++---------
 drivers/hv/hv_utils_transport.h |  1 +
 2 files changed, 56 insertions(+), 14 deletions(-)

diff --git a/drivers/hv/hv_utils_transport.c b/drivers/hv/hv_utils_transport.c
index 5cce2d2..984cba5 100644
--- a/drivers/hv/hv_utils_transport.c
+++ b/drivers/hv/hv_utils_transport.c
@@ -27,11 +27,9 @@ static struct list_head hvt_list = LIST_HEAD_INIT(hvt_list);
 
 static void hvt_reset(struct hvutil_transport *hvt)
 {
-	mutex_lock(&hvt->lock);
 	kfree(hvt->outmsg);
 	hvt->outmsg = NULL;
 	hvt->outmsg_len = 0;
-	mutex_unlock(&hvt->lock);
 	if (hvt->on_reset)
 		hvt->on_reset();
 }
@@ -44,10 +42,17 @@ static ssize_t hvt_op_read(struct file *file, char __user *buf,
 
 	hvt = container_of(file->f_op, struct hvutil_transport, fops);
 
-	if (wait_event_interruptible(hvt->outmsg_q, hvt->outmsg_len > 0))
+	if (wait_event_interruptible(hvt->outmsg_q, hvt->outmsg_len > 0 ||
+				     hvt->mode != HVUTIL_TRANSPORT_CHARDEV))
 		return -EINTR;
 
 	mutex_lock(&hvt->lock);
+
+	if (hvt->mode == HVUTIL_TRANSPORT_DESTROY) {
+		ret = -EBADF;
+		goto out_unlock;
+	}
+
 	if (!hvt->outmsg) {
 		ret = -EAGAIN;
 		goto out_unlock;
@@ -85,6 +90,9 @@ static ssize_t hvt_op_write(struct file *file, const char __user *buf,
 	if (IS_ERR(inmsg))
 		return PTR_ERR(inmsg);
 
+	if (hvt->mode == HVUTIL_TRANSPORT_DESTROY)
+		return -EBADF;
+
 	if (hvt->on_msg(inmsg, count))
 		ret = -EFAULT;
 	kfree(inmsg);
@@ -99,6 +107,10 @@ static unsigned int hvt_op_poll(struct file *file, poll_table *wait)
 	hvt = container_of(file->f_op, struct hvutil_transport, fops);
 
 	poll_wait(file, &hvt->outmsg_q, wait);
+
+	if (hvt->mode == HVUTIL_TRANSPORT_DESTROY)
+		return -EBADF;
+
 	if (hvt->outmsg_len > 0)
 		return POLLIN | POLLRDNORM;
 
@@ -108,26 +120,39 @@ static unsigned int hvt_op_poll(struct file *file, poll_table *wait)
 static int hvt_op_open(struct inode *inode, struct file *file)
 {
 	struct hvutil_transport *hvt;
+	int ret = 0;
+	bool issue_reset = false;
 
 	hvt = container_of(file->f_op, struct hvutil_transport, fops);
 
-	/*
-	 * Switching to CHARDEV mode. We switch bach to INIT when device
-	 * gets released.
-	 */
-	if (hvt->mode == HVUTIL_TRANSPORT_INIT)
+	mutex_lock(&hvt->lock);
+
+	if (hvt->mode == HVUTIL_TRANSPORT_DESTROY) {
+		ret = -EBADF;
+	} else if (hvt->mode == HVUTIL_TRANSPORT_INIT) {
+		/*
+		 * Switching to CHARDEV mode. We switch bach to INIT when
+		 * device gets released.
+		 */
 		hvt->mode = HVUTIL_TRANSPORT_CHARDEV;
+	}
 	else if (hvt->mode == HVUTIL_TRANSPORT_NETLINK) {
 		/*
 		 * We're switching from netlink communication to using char
 		 * device. Issue the reset first.
 		 */
-		hvt_reset(hvt);
+		issue_reset = true;
 		hvt->mode = HVUTIL_TRANSPORT_CHARDEV;
-	} else
-		return -EBUSY;
+	} else {
+		ret = -EBUSY;
+	}
 
-	return 0;
+	if (issue_reset)
+		hvt_reset(hvt);
+
+	mutex_unlock(&hvt->lock);
+
+	return ret;
 }
 
 static int hvt_op_release(struct inode *inode, struct file *file)
@@ -136,12 +161,15 @@ static int hvt_op_release(struct inode *inode, struct file *file)
 
 	hvt = container_of(file->f_op, struct hvutil_transport, fops);
 
-	hvt->mode = HVUTIL_TRANSPORT_INIT;
+	mutex_lock(&hvt->lock);
+	if (hvt->mode != HVUTIL_TRANSPORT_DESTROY)
+		hvt->mode = HVUTIL_TRANSPORT_INIT;
 	/*
 	 * Cleanup message buffers to avoid spurious messages when the daemon
 	 * connects back.
 	 */
 	hvt_reset(hvt);
+	mutex_unlock(&hvt->lock);
 
 	return 0;
 }
@@ -168,6 +196,7 @@ static void hvt_cn_callback(struct cn_msg *msg, struct netlink_skb_parms *nsp)
 	 * Switching to NETLINK mode. Switching to CHARDEV happens when someone
 	 * opens the device.
 	 */
+	mutex_lock(&hvt->lock);
 	if (hvt->mode == HVUTIL_TRANSPORT_INIT)
 		hvt->mode = HVUTIL_TRANSPORT_NETLINK;
 
@@ -175,6 +204,7 @@ static void hvt_cn_callback(struct cn_msg *msg, struct netlink_skb_parms *nsp)
 		hvt_found->on_msg(msg->data, msg->len);
 	else
 		pr_warn("hvt_cn_callback: unexpected netlink message!\n");
+	mutex_unlock(&hvt->lock);
 }
 
 int hvutil_transport_send(struct hvutil_transport *hvt, void *msg, int len)
@@ -182,7 +212,8 @@ int hvutil_transport_send(struct hvutil_transport *hvt, void *msg, int len)
 	struct cn_msg *cn_msg;
 	int ret = 0;
 
-	if (hvt->mode == HVUTIL_TRANSPORT_INIT) {
+	if (hvt->mode == HVUTIL_TRANSPORT_INIT ||
+	    hvt->mode == HVUTIL_TRANSPORT_DESTROY) {
 		return -EINVAL;
 	} else if (hvt->mode == HVUTIL_TRANSPORT_NETLINK) {
 		cn_msg = kzalloc(sizeof(*cn_msg) + len, GFP_ATOMIC);
@@ -198,6 +229,11 @@ int hvutil_transport_send(struct hvutil_transport *hvt, void *msg, int len)
 	}
 	/* HVUTIL_TRANSPORT_CHARDEV */
 	mutex_lock(&hvt->lock);
+	if (hvt->mode != HVUTIL_TRANSPORT_CHARDEV) {
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
 	if (hvt->outmsg) {
 		/* Previous message wasn't received */
 		ret = -EFAULT;
@@ -268,6 +304,11 @@ err_free_hvt:
 
 void hvutil_transport_destroy(struct hvutil_transport *hvt)
 {
+	mutex_lock(&hvt->lock);
+	hvt->mode = HVUTIL_TRANSPORT_DESTROY;
+	wake_up_interruptible(&hvt->outmsg_q);
+	mutex_unlock(&hvt->lock);
+
 	spin_lock(&hvt_list_lock);
 	list_del(&hvt->list);
 	spin_unlock(&hvt_list_lock);
diff --git a/drivers/hv/hv_utils_transport.h b/drivers/hv/hv_utils_transport.h
index bff4c92..06254a1 100644
--- a/drivers/hv/hv_utils_transport.h
+++ b/drivers/hv/hv_utils_transport.h
@@ -25,6 +25,7 @@ enum hvutil_transport_mode {
 	HVUTIL_TRANSPORT_INIT = 0,
 	HVUTIL_TRANSPORT_NETLINK,
 	HVUTIL_TRANSPORT_CHARDEV,
+	HVUTIL_TRANSPORT_DESTROY,
 };
 
 struct hvutil_transport {
-- 
2.4.3

[PATCH 4/4] Drivers: hv: utils: fix crash when device is removed from host side

[Vitaly Kuznetsov]

The crash is observed when a service is being disabled host side while
userspace daemon is connected to the device:

[   90.244859] general protection fault: 0000 [#1] SMP
...
[   90.800082] Call Trace:
[   90.800082]  [<ffffffff81187008>] __fput+0xc8/0x1f0
[   90.800082]  [<ffffffff8118716e>] ____fput+0xe/0x10
...
[   90.800082]  [<ffffffff81015278>] do_signal+0x28/0x580
[   90.800082]  [<ffffffff81086656>] ? finish_task_switch+0xa6/0x180
[   90.800082]  [<ffffffff81443ebf>] ? __schedule+0x28f/0x870
[   90.800082]  [<ffffffffa01ebbaa>] ? hvt_op_read+0x12a/0x140 [hv_utils]
...

The problem is that hvutil_transport_destroy() which does misc_deregister()
freeing the appropriate device is reachable by two paths: module unload
and from util_remove(). While module unload path is protected by .owner in
struct file_operations util_remove() path is not. Freeing the device while
someone holds an open fd for it is a show stopper.

In general, it is not possible to revoke an fd from all users so the only
way to solve the issue is to defer freeing the hvutil_transport structure.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 drivers/hv/hv_utils_transport.c | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)

diff --git a/drivers/hv/hv_utils_transport.c b/drivers/hv/hv_utils_transport.c
index 984cba5..174c72a 100644
--- a/drivers/hv/hv_utils_transport.c
+++ b/drivers/hv/hv_utils_transport.c
@@ -155,13 +155,22 @@ static int hvt_op_open(struct inode *inode, struct file *file)
 	return ret;
 }
 
+static void hvt_transport_free(struct hvutil_transport *hvt)
+{
+	misc_deregister(&hvt->mdev);
+	kfree(hvt->outmsg);
+	kfree(hvt);
+}
+
 static int hvt_op_release(struct inode *inode, struct file *file)
 {
 	struct hvutil_transport *hvt;
+	int mode_old;
 
 	hvt = container_of(file->f_op, struct hvutil_transport, fops);
 
 	mutex_lock(&hvt->lock);
+	mode_old = hvt->mode;
 	if (hvt->mode != HVUTIL_TRANSPORT_DESTROY)
 		hvt->mode = HVUTIL_TRANSPORT_INIT;
 	/*
@@ -171,6 +180,9 @@ static int hvt_op_release(struct inode *inode, struct file *file)
 	hvt_reset(hvt);
 	mutex_unlock(&hvt->lock);
 
+	if (mode_old == HVUTIL_TRANSPORT_DESTROY)
+		hvt_transport_free(hvt);
+
 	return 0;
 }
 
@@ -304,17 +316,25 @@ err_free_hvt:
 
 void hvutil_transport_destroy(struct hvutil_transport *hvt)
 {
+	int mode_old;
+
 	mutex_lock(&hvt->lock);
+	mode_old = hvt->mode;
 	hvt->mode = HVUTIL_TRANSPORT_DESTROY;
 	wake_up_interruptible(&hvt->outmsg_q);
 	mutex_unlock(&hvt->lock);
 
+	/*
+	 * In case we were in 'chardev' mode we still have an open fd so we
+	 * have to defer freeing the device. Netlink interface can be freed
+	 * now.
+	 */
 	spin_lock(&hvt_list_lock);
 	list_del(&hvt->list);
 	spin_unlock(&hvt_list_lock);
 	if (hvt->cn_id.idx > 0 && hvt->cn_id.val > 0)
 		cn_del_callback(&hvt->cn_id);
-	misc_deregister(&hvt->mdev);
-	kfree(hvt->outmsg);
-	kfree(hvt);
+
+	if (mode_old != HVUTIL_TRANSPORT_CHARDEV)
+		hvt_transport_free(hvt);
 }
-- 
2.4.3