Re: deadlock between tty_write and tty_send_xchar

[Peter Hurley <peter@hurleysoftware.com>]

On 11/11/2015 03:35 AM, Dmitry Vyukov wrote:
> Hello,
> 
> I've hit the following deadlock while running syzkaller on commit
> ce5c2d2c256a4c8b523036537cd6be2d6af8f69d (Nov 7):
> 
> [ INFO: possible circular locking dependency detected ]
> 4.3.0+ #57 Not tainted
> -------------------------------------------------------
> syzkaller_execu/24882 is trying to acquire lock:
>  (&tty->atomic_write_lock){+.+.+.}, at: [<ffffffff81774876>]
> tty_write_lock+0x46/0x70 drivers/tty/tty_io.c:1093
> 
> but task is already holding lock:
>  (&tty->termios_rwsem){++++.+}, at: [<ffffffff817864d7>]
> n_tty_ioctl_helper+0x177/0x210 drivers/tty/tty_ioctl.c:1150
> 
> which lock already depends on the new lock.
> 
> the existing dependency chain (in reverse order) is:
> 
> -> #1 (&tty->termios_rwsem){++++.+}:
>        [<ffffffff81122501>] lock_acquire+0x101/0x1d0 kernel/locking/lockdep.c:3585
>        [<ffffffff821727a9>] down_read+0x39/0x50 kernel/locking/rwsem.c:22
>        [<ffffffff8177eb07>] n_tty_write+0x137/0xaa0 drivers/tty/n_tty.c:2362
>        [<     inline     >] do_tty_write drivers/tty/tty_io.c:1164
>        [<ffffffff8177832e>] tty_write+0x28e/0x4f0 drivers/tty/tty_io.c:1248
>        [<ffffffff81778631>] redirected_tty_write+0xa1/0xb0 drivers/tty/tty_io.c:1269
>        [<ffffffff812d788b>] __vfs_write+0xeb/0x2a0 fs/read_write.c:489
>        [<ffffffff812d8a63>] vfs_write+0x113/0x290 fs/read_write.c:538
>        [<     inline     >] SYSC_write fs/read_write.c:585
>        [<ffffffff812da27b>] SyS_write+0xbb/0x170 fs/read_write.c:577
>        [<ffffffff82175fd1>] entry_SYSCALL_64_fastpath+0x31/0x9a
> arch/x86/entry/entry_64.S:187
> 
> -> #0 (&tty->atomic_write_lock){+.+.+.}:
>        [<     inline     >] __mutex_lock_common kernel/locking/mutex.c:518
>        [<ffffffff82170525>] mutex_lock_interruptible_nested+0xa5/0x660 kernel/locking/mutex.c:647
>        [<ffffffff81774876>] tty_write_lock+0x46/0x70 drivers/tty/tty_io.c:1093
>        [<ffffffff8177a5a4>] tty_send_xchar+0x94/0x130 drivers/tty/tty_io.c:1289
>        [<ffffffff81786507>] n_tty_ioctl_helper+0x1a7/0x210 drivers/tty/tty_ioctl.c:1158
>        [<ffffffff8177f6e9>] n_tty_ioctl+0xe9/0x1e0 drivers/tty/n_tty.c:2514
>        [<ffffffff8177979c>] tty_ioctl+0xa4c/0x1650 drivers/tty/tty_io.c:2945
>        [<     inline     >] vfs_ioctl fs/ioctl.c:43
>        [<ffffffff812fdf0f>] do_vfs_ioctl+0x53f/0x980 fs/ioctl.c:607
>        [<     inline     >] SYSC_ioctl fs/ioctl.c:622
>        [<ffffffff812fe3df>] SyS_ioctl+0x8f/0xc0 fs/ioctl.c:613
>        [<ffffffff82175fd1>] entry_SYSCALL_64_fastpath+0x31/0x9a
> arch/x86/entry/entry_64.S:187
> 
> other info that might help us debug this:
> 
>  Possible unsafe locking scenario:
> 
>        CPU0                    CPU1
>        ----                    ----
>   lock(&tty->termios_rwsem);
>                                lock(&tty->atomic_write_lock);
>                                lock(&tty->termios_rwsem);
>   lock(&tty->atomic_write_lock);
> 
>  *** DEADLOCK ***

Thanks for the report, Dmitry.
Please re-test with the accompanying patch.

Regards,
Peter Hurley

--- >% ---
Subject: [PATCH] tty: Fix tty_send_xchar() lock order inversion

The correct lock order is atomic_write_lock => termios_rwsem, as
established by tty_write() => n_tty_write().

Fixes: c274f6ef1c666 ("tty: Hold termios_rwsem for tcflow(TCIxxx)")
Reported-by: Dmitry Vyukov <dvyukov@google.com>
Cc: <stable@vger.kernel.org> # v3.18+
Signed-off-by: Peter Hurley <peter@hurleysoftware.com>
---
 drivers/tty/tty_io.c    | 4 ++++
 drivers/tty/tty_ioctl.c | 4 ----
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/tty/tty_io.c b/drivers/tty/tty_io.c
index 2f8c21e..346a1a5 100644
--- a/drivers/tty/tty_io.c
+++ b/drivers/tty/tty_io.c
@@ -1283,18 +1283,22 @@ int tty_send_xchar(struct tty_struct *tty, char ch)
 	int	was_stopped = tty->stopped;
 
 	if (tty->ops->send_xchar) {
+		down_read(&tty->termios_rwsem);
 		tty->ops->send_xchar(tty, ch);
+		up_read(&tty->termios_rwsem);
 		return 0;
 	}
 
 	if (tty_write_lock(tty, 0) < 0)
 		return -ERESTARTSYS;
 
+	down_read(&tty->termios_rwsem);
 	if (was_stopped)
 		start_tty(tty);
 	tty->ops->write(tty, &ch, 1);
 	if (was_stopped)
 		stop_tty(tty);
+	up_read(&tty->termios_rwsem);
 	tty_write_unlock(tty);
 	return 0;
 }
diff --git a/drivers/tty/tty_ioctl.c b/drivers/tty/tty_ioctl.c
index b8c5c12..0ea3513 100644
--- a/drivers/tty/tty_ioctl.c
+++ b/drivers/tty/tty_ioctl.c
@@ -1140,16 +1140,12 @@ int n_tty_ioctl_helper(struct tty_struct *tty, struct file *file,
 			spin_unlock_irq(&tty->flow_lock);
 			break;
 		case TCIOFF:
-			down_read(&tty->termios_rwsem);
 			if (STOP_CHAR(tty) != __DISABLED_CHAR)
 				retval = tty_send_xchar(tty, STOP_CHAR(tty));
-			up_read(&tty->termios_rwsem);
 			break;
 		case TCION:
-			down_read(&tty->termios_rwsem);
 			if (START_CHAR(tty) != __DISABLED_CHAR)
 				retval = tty_send_xchar(tty, START_CHAR(tty));
-			up_read(&tty->termios_rwsem);
 			break;
 		default:
 			return -EINVAL;
-- 
2.6.3