The Linux Kernel Mailing List
 help / color / mirror / Atom feed
From: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
To: Steven Rostedt <rostedt@goodmis.org>
Cc: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>,
	bugzilla-daemon@bugzilla.kernel.org,
	LKML <linux-kernel@vger.kernel.org>,
	wen.yang99@zte.com.cn, Petr Mladek <pmladek@suse.com>,
	Peter Zijlstra <peterz@infradead.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	Christoph Hellwig <hch@infradead.org>,
	Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Subject: Re: [Bug 199003] console stalled, cause Hard LOCKUP.
Date: Thu, 22 Mar 2018 11:34:57 +0900	[thread overview]
Message-ID: <20180322023457.GB3181@jagdpanzerIV> (raw)
In-Reply-To: <20180322021437.GA3181@jagdpanzerIV>

On (03/22/18 11:14), Sergey Senozhatsky wrote:
[..]
> Looking at
>  printk()->call_console_drivers()->serial8250_console_putchar()->wait_for_xmitr()
> 
> ... wait_for_xmitr() can spin for over 1 second waiting for the UART_MSR_CTS
> bit.

[..]

> a 1+ second long busy loop in the console driver is quite close to
> "problems guaranteed". But, wait, there is even more. This wait_for_xmitr()
> busy wait is happening after every character we print on the console. So
> printk("foo") will generate 5 * wait_for_xmitr() busy loops [foo + \r + \n].
> They punch&touch watchdog a lot, so at the least the system won't get killed
> by the hardlockup detector. But at the same time, it's still potentially a
> 1+ second busy loop in the console driver * strlen(message).

One does not even need to have concurrent printk()-s in this case. A
single CPU doing several direct printks under spin_lock is already
enough:

	CPUA						CPUB ~ CPUZ
	spin_lock(&lock)
	 printk->wait_for_xmitr				spin_lock(&lock)
	 printk->wait_for_xmitr
	 ...
	 printk->wait_for_xmitr				<< lockups >>
	 printk->wait_for_xmitr
	spin_unlock(&lock)

> Sometimes I really wish we had detached consoles. Direct printk()->console
> is nice and cool, but... we can't have it.

And this is, basically, what they do with printk_deferred(). We usually
use it to avoid deadlocks, but in this particular case it's used due to
the fact that direct printk() is way too painful, so they are detaching
printout and move it to another control path. Quite an interesting idea,
I must say.

	-ss

  reply	other threads:[~2018-03-22  2:35 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <bug-199003-8243@https.bugzilla.kernel.org/>
     [not found] ` <bug-199003-8243-Txkdkdv5Cm@https.bugzilla.kernel.org/>
2018-03-21 13:44   ` [Bug 199003] console stalled, cause Hard LOCKUP Steven Rostedt
2018-03-22  2:14     ` Sergey Senozhatsky
2018-03-22  2:34       ` Sergey Senozhatsky [this message]
2018-03-22 22:25       ` Steven Rostedt
2018-03-23 12:06         ` Sergey Senozhatsky
2018-03-23 13:16           ` Petr Mladek
2018-03-26  5:12             ` Sergey Senozhatsky
2018-03-26  9:26               ` Petr Mladek
     [not found] <bug-199003-14532@https.bugzilla.kernel.org/>
     [not found] ` <bug-199003-14532-lzL5ySZS5x@https.bugzilla.kernel.org/>
2018-03-05  9:27   ` Sergey Senozhatsky
     [not found] ` <bug-199003-14532-4aWzZJgGHz@https.bugzilla.kernel.org/>
2018-03-21  7:28   ` Sergey Senozhatsky
     [not found] ` <bug-199003-14532-DgJdGAHEOE@https.bugzilla.kernel.org/>
2018-03-26  5:18   ` Sergey Senozhatsky
     [not found] ` <bug-199003-14532-IBTfRpvy7t@https.bugzilla.kernel.org/>
2018-03-27 10:37   ` Sergey Senozhatsky

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180322023457.GB3181@jagdpanzerIV \
    --to=sergey.senozhatsky.work@gmail.com \
    --cc=akpm@linux-foundation.org \
    --cc=bugzilla-daemon@bugzilla.kernel.org \
    --cc=hch@infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=peterz@infradead.org \
    --cc=pmladek@suse.com \
    --cc=rostedt@goodmis.org \
    --cc=sergey.senozhatsky@gmail.com \
    --cc=wen.yang99@zte.com.cn \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox