From: Peter Hurley <peter@hurleysoftware.com>
To: Peter Zijlstra <peterz@infradead.org>,
Dmitry Vyukov <dvyukov@google.com>, Jiri Slaby <jslaby@suse.com>,
gnomes@lxorguk.ukuu.org.uk
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
LKML <linux-kernel@vger.kernel.org>,
J Freyensee <james_p_freyensee@linux.intel.com>,
syzkaller <syzkaller@googlegroups.com>,
Kostya Serebryany <kcc@google.com>,
Alexander Potapenko <glider@google.com>,
Sasha Levin <sasha.levin@oracle.com>,
Eric Dumazet <edumazet@google.com>
Subject: Re: tty: deadlock between n_tracerouter_receivebuf and flush_to_ldisc
Date: Wed, 20 Jan 2016 08:08:47 -0800 [thread overview]
Message-ID: <569FB10F.4080205@hurleysoftware.com> (raw)
In-Reply-To: <20160120130201.GA1027@worktop>
On 01/20/2016 05:02 AM, Peter Zijlstra wrote:
> On Wed, Dec 30, 2015 at 11:44:01AM +0100, Dmitry Vyukov wrote:
>> -> #3 (&buf->lock){+.+...}:
>> [<ffffffff813f0acf>] lock_acquire+0x19f/0x3c0 kernel/locking/lockdep.c:3585
>> [< inline >] __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:112
>> [<ffffffff85c8e790>] _raw_spin_lock_irqsave+0x50/0x70 kernel/locking/spinlock.c:159
>> [<ffffffff82b8c050>] tty_get_pgrp+0x20/0x80 drivers/tty/tty_io.c:2502
>
> So in any recent code that I look at this function tries to acquire
> tty->ctrl_lock, not buf->lock. Am I missing something ?!
Yes.
The tty locks were annotated with __lockfunc so were being elided from lockdep
stacktraces. Greg has a patch in his queue from me that removes the __lockfunc
annotation ("tty: Remove __lockfunc annotation from tty lock functions").
Unfortunately, I think syzkaller's post-processing stack trace isn't helping
either, giving the impression that the stack is still inside tty_get_pgrp().
It's not.
It's in pty_flush_buffer(), which is taking the 'other tty' buf->lock.
Looks to me like the lock inversion is caused by the tty_driver_flush_buffer()
in n_tracerouter_open()/_close(), but I need to look at this mess a little
closer.
Regards,
Peter Hurley
>> [<ffffffff82b9a09a>] __isig+0x1a/0x50 drivers/tty/n_tty.c:1112
>> [<ffffffff82b9c16e>] isig+0xae/0x2c0 drivers/tty/n_tty.c:1131
>> [<ffffffff82b9ef02>] n_tty_receive_signal_char+0x22/0xf0 drivers/tty/n_tty.c:1243
>> [<ffffffff82ba4958>] n_tty_receive_char_special+0x1278/0x2bf0 drivers/tty/n_tty.c:1289
>> [< inline >] n_tty_receive_buf_fast drivers/tty/n_tty.c:1613
>> [< inline >] __receive_buf drivers/tty/n_tty.c:1647
>> [<ffffffff82ba7ca6>] n_tty_receive_buf_common+0x19d6/0x2450 drivers/tty/n_tty.c:1745
>> [<ffffffff82ba8753>] n_tty_receive_buf2+0x33/0x40 drivers/tty/n_tty.c:1780
>> [< inline >] receive_buf drivers/tty/tty_buffer.c:450
>> [<ffffffff82bafa6f>] flush_to_ldisc+0x3bf/0x7f0 drivers/tty/tty_buffer.c:517
>> [<ffffffff8133833c>] process_one_work+0x76c/0x13e0 kernel/workqueue.c:2030
>> [<ffffffff81339093>] worker_thread+0xe3/0xe90 kernel/workqueue.c:2162
>> [<ffffffff8134b63f>] kthread+0x23f/0x2d0 drivers/block/aoe/aoecmd.c:1303
>> [<ffffffff85c8eeef>] ret_from_fork+0x3f/0x70 arch/x86/entry/entry_64.S:468
next prev parent reply other threads:[~2016-01-20 16:08 UTC|newest]
Thread overview: 31+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-12-30 10:44 tty: deadlock between n_tracerouter_receivebuf and flush_to_ldisc Dmitry Vyukov
2016-01-15 7:51 ` Dmitry Vyukov
2016-01-15 16:33 ` One Thousand Gnomes
2016-01-15 17:22 ` Dmitry Vyukov
2016-01-20 9:36 ` Dmitry Vyukov
2016-01-20 11:44 ` Peter Zijlstra
2016-01-20 11:54 ` Dmitry Vyukov
2016-01-20 12:07 ` Peter Zijlstra
2016-01-20 14:58 ` One Thousand Gnomes
2016-01-20 15:16 ` Dmitry Vyukov
2016-01-20 16:32 ` Peter Zijlstra
2016-01-20 2:09 ` J Freyensee
2016-01-20 12:47 ` Jiri Slaby
2016-01-20 13:02 ` Peter Zijlstra
2016-01-20 13:07 ` Dmitry Vyukov
2016-01-20 16:08 ` Peter Hurley [this message]
2016-01-20 20:47 ` Peter Hurley
2016-01-21 10:06 ` Dmitry Vyukov
2016-01-21 10:20 ` Peter Zijlstra
2016-01-21 17:51 ` Peter Hurley
2016-01-22 14:10 ` Dmitry Vyukov
2016-01-25 16:56 ` Peter Hurley
2016-01-21 17:43 ` Peter Hurley
2016-02-03 4:24 ` Peter Hurley
2016-02-03 17:32 ` Dmitry Vyukov
2016-02-03 19:09 ` Peter Hurley
2016-02-04 12:39 ` Dmitry Vyukov
2016-02-04 13:17 ` Dmitry Vyukov
2016-02-04 18:46 ` Peter Hurley
2016-02-04 18:48 ` Dmitry Vyukov
2016-02-05 21:22 ` Dmitry Vyukov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=569FB10F.4080205@hurleysoftware.com \
--to=peter@hurleysoftware.com \
--cc=dvyukov@google.com \
--cc=edumazet@google.com \
--cc=glider@google.com \
--cc=gnomes@lxorguk.ukuu.org.uk \
--cc=gregkh@linuxfoundation.org \
--cc=james_p_freyensee@linux.intel.com \
--cc=jslaby@suse.com \
--cc=kcc@google.com \
--cc=linux-kernel@vger.kernel.org \
--cc=peterz@infradead.org \
--cc=sasha.levin@oracle.com \
--cc=syzkaller@googlegroups.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).