From: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
To: Tejun Heo <tj@kernel.org>, Peter Zijlstra <peterz@infradead.org>
Cc: Petr Mladek <pmladek@suse.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
akpm@linux-foundation.org, Steven Rostedt <rostedt@goodmis.org>,
Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
linux-mm@kvack.org, Cong Wang <xiyou.wangcong@gmail.com>,
Dave Hansen <dave.hansen@intel.com>,
Johannes Weiner <hannes@cmpxchg.org>,
Mel Gorman <mgorman@suse.de>, Michal Hocko <mhocko@kernel.org>,
Vlastimil Babka <vbabka@suse.cz>, Jan Kara <jack@suse.cz>,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>,
rostedt@home.goodmis.org, Byungchul Park <byungchul.park@lge.com>,
Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>,
Pavel Machek <pavel@ucw.cz>,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH v5 0/2] printk: Console owner and waiter logic cleanup
Date: Thu, 11 Jan 2018 14:15:24 +0900 [thread overview]
Message-ID: <20180111051524.GC494@jagdpanzerIV> (raw)
In-Reply-To: <20180110182153.GP6176@hirez.programming.kicks-ass.net>
On (01/10/18 19:21), Peter Zijlstra wrote:
>
> On Wed, Jan 10, 2018 at 09:02:23AM -0800, Tejun Heo wrote:
> > 2. System runs out of memory, OOM triggers.
> > 3. OOM handler is printing out OOM debug info.
> > 4. While trying to emit the messages for netconsole, the network stack
> > / driver tries to allocate memory and then fail, which in turn
> > triggers allocation failure or other warning messages. printk was
> > already flushing, so the messages are queued on the ring.
> > 5. OOM handler keeps flushing but 4 repeats and the queue is never
> > shrinking. Because OOM handler is trapped in printk flushing, it
> > never manages to free memory and no one else can enter OOM path
> > either, so the system is trapped in this state.
>
> Why not kill recursive OOM (msgs) ?
hm... do I understand it correctly that there is a
console_unlock()->call_console_drivers()->FOO_write()->kmalloc()->printk() recursion?
we call console drivers from printk-safe context now. so those printks
from kmalloc are redirected to per-CPU printk-safe buffer, which is
limited in size (we probably might start losing some of those OOM
messages) and which is flushed (log_store()) from another context.
-ss
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
WARNING: multiple messages have this Message-ID (diff)
From: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
To: Tejun Heo <tj@kernel.org>, Peter Zijlstra <peterz@infradead.org>
Cc: Petr Mladek <pmladek@suse.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
akpm@linux-foundation.org, Steven Rostedt <rostedt@goodmis.org>,
Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
linux-mm@kvack.org, Cong Wang <xiyou.wangcong@gmail.com>,
Dave Hansen <dave.hansen@intel.com>,
Johannes Weiner <hannes@cmpxchg.org>,
Mel Gorman <mgorman@suse.de>, Michal Hocko <mhocko@kernel.org>,
Vlastimil Babka <vbabka@suse.cz>, Jan Kara <jack@suse.cz>,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>,
rostedt@home.goodmis.org, Byungchul Park <byungchul.park@lge.com>,
Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>,
Pavel Machek <pavel@ucw.cz>,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH v5 0/2] printk: Console owner and waiter logic cleanup
Date: Thu, 11 Jan 2018 14:15:24 +0900 [thread overview]
Message-ID: <20180111051524.GC494@jagdpanzerIV> (raw)
In-Reply-To: <20180110182153.GP6176@hirez.programming.kicks-ass.net>
On (01/10/18 19:21), Peter Zijlstra wrote:
>
> On Wed, Jan 10, 2018 at 09:02:23AM -0800, Tejun Heo wrote:
> > 2. System runs out of memory, OOM triggers.
> > 3. OOM handler is printing out OOM debug info.
> > 4. While trying to emit the messages for netconsole, the network stack
> > / driver tries to allocate memory and then fail, which in turn
> > triggers allocation failure or other warning messages. printk was
> > already flushing, so the messages are queued on the ring.
> > 5. OOM handler keeps flushing but 4 repeats and the queue is never
> > shrinking. Because OOM handler is trapped in printk flushing, it
> > never manages to free memory and no one else can enter OOM path
> > either, so the system is trapped in this state.
>
> Why not kill recursive OOM (msgs) ?
hm... do I understand it correctly that there is a
console_unlock()->call_console_drivers()->FOO_write()->kmalloc()->printk() recursion?
we call console drivers from printk-safe context now. so those printks
from kmalloc are redirected to per-CPU printk-safe buffer, which is
limited in size (we probably might start losing some of those OOM
messages) and which is flushed (log_store()) from another context.
-ss
next prev parent reply other threads:[~2018-01-11 5:15 UTC|newest]
Thread overview: 280+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-01-10 13:24 [PATCH v5 0/2] printk: Console owner and waiter logic cleanup Petr Mladek
2018-01-10 13:24 ` Petr Mladek
2018-01-10 13:24 ` [PATCH v5 1/2] printk: Add console owner and waiter logic to load balance console writes Petr Mladek
2018-01-10 13:24 ` Petr Mladek
2018-01-10 16:50 ` Steven Rostedt
2018-01-10 16:50 ` Steven Rostedt
2018-01-12 16:54 ` Steven Rostedt
2018-01-12 16:54 ` Steven Rostedt
2018-01-12 17:11 ` Steven Rostedt
2018-01-12 17:11 ` Steven Rostedt
2018-01-17 19:13 ` Rasmus Villemoes
2018-01-17 19:13 ` Rasmus Villemoes
2018-01-17 19:33 ` Steven Rostedt
2018-01-17 19:33 ` Steven Rostedt
2018-01-19 9:51 ` Sergey Senozhatsky
2018-01-19 9:51 ` Sergey Senozhatsky
2018-01-18 22:03 ` Pavel Machek
2018-01-18 22:03 ` Pavel Machek
2018-01-19 0:20 ` Steven Rostedt
2018-01-19 0:20 ` Steven Rostedt
2018-01-17 2:19 ` Byungchul Park
2018-01-17 2:19 ` Byungchul Park
2018-01-17 4:54 ` Byungchul Park
2018-01-17 4:54 ` Byungchul Park
2018-01-17 7:34 ` Byungchul Park
2018-01-17 7:34 ` Byungchul Park
2018-01-17 12:04 ` Petr Mladek
2018-01-17 12:04 ` Petr Mladek
2018-01-18 1:53 ` Byungchul Park
2018-01-18 1:53 ` Byungchul Park
2018-01-18 1:57 ` Byungchul Park
2018-01-18 1:57 ` Byungchul Park
2018-01-18 2:19 ` Steven Rostedt
2018-01-18 2:19 ` Steven Rostedt
2018-01-18 4:01 ` Byungchul Park
2018-01-18 4:01 ` Byungchul Park
2018-01-18 15:21 ` Steven Rostedt
2018-01-18 15:21 ` Steven Rostedt
2018-01-19 2:37 ` Byungchul Park
2018-01-19 2:37 ` Byungchul Park
2018-01-19 3:27 ` Steven Rostedt
2018-01-19 3:27 ` Steven Rostedt
2018-01-22 2:31 ` Byungchul Park
2018-01-22 2:31 ` Byungchul Park
2018-01-10 13:24 ` [PATCH v5 2/2] printk: Hide console waiter logic into helpers Petr Mladek
2018-01-10 13:24 ` Petr Mladek
2018-01-10 17:52 ` Steven Rostedt
2018-01-10 17:52 ` Steven Rostedt
2018-01-11 12:03 ` Petr Mladek
2018-01-11 12:03 ` Petr Mladek
2018-01-12 15:37 ` Steven Rostedt
2018-01-12 15:37 ` Steven Rostedt
2018-01-12 16:08 ` Petr Mladek
2018-01-12 16:08 ` Petr Mladek
2018-01-12 16:36 ` Steven Rostedt
2018-01-12 16:36 ` Steven Rostedt
2018-01-15 16:08 ` Petr Mladek
2018-01-15 16:08 ` Petr Mladek
2018-01-16 5:05 ` Sergey Senozhatsky
2018-01-16 5:05 ` Sergey Senozhatsky
2018-01-10 14:05 ` [PATCH v5 0/2] printk: Console owner and waiter logic cleanup Tejun Heo
2018-01-10 14:05 ` Tejun Heo
2018-01-10 16:29 ` Petr Mladek
2018-01-10 16:29 ` Petr Mladek
2018-01-10 17:02 ` Tejun Heo
2018-01-10 17:02 ` Tejun Heo
2018-01-10 18:21 ` Peter Zijlstra
2018-01-10 18:21 ` Peter Zijlstra
2018-01-10 18:30 ` Tejun Heo
2018-01-10 18:30 ` Tejun Heo
2018-01-10 18:41 ` Peter Zijlstra
2018-01-10 18:41 ` Peter Zijlstra
2018-01-10 19:05 ` Tejun Heo
2018-01-10 19:05 ` Tejun Heo
2018-01-11 5:15 ` Sergey Senozhatsky [this message]
2018-01-11 5:15 ` Sergey Senozhatsky
2018-01-10 18:22 ` Steven Rostedt
2018-01-10 18:22 ` Steven Rostedt
2018-01-10 18:36 ` Tejun Heo
2018-01-10 18:36 ` Tejun Heo
2018-01-10 18:40 ` Mathieu Desnoyers
2018-01-10 18:40 ` Mathieu Desnoyers
2018-01-11 7:36 ` Sergey Senozhatsky
2018-01-11 7:36 ` Sergey Senozhatsky
2018-01-11 11:24 ` Petr Mladek
2018-01-11 11:24 ` Petr Mladek
2018-01-11 13:19 ` Sergey Senozhatsky
2018-01-11 13:19 ` Sergey Senozhatsky
2018-01-24 9:36 ` Peter Zijlstra
2018-01-24 9:36 ` Peter Zijlstra
2018-01-24 18:46 ` Tejun Heo
2018-01-24 18:46 ` Tejun Heo
2018-05-09 8:58 ` Sergey Senozhatsky
2018-05-09 8:58 ` Sergey Senozhatsky
2018-01-10 18:54 ` Steven Rostedt
2018-01-10 18:54 ` Steven Rostedt
2018-01-11 5:10 ` Sergey Senozhatsky
2018-01-11 5:10 ` Sergey Senozhatsky
2018-01-10 18:05 ` Steven Rostedt
2018-01-10 18:05 ` Steven Rostedt
2018-01-10 18:12 ` Tejun Heo
2018-01-10 18:12 ` Tejun Heo
2018-01-10 18:14 ` Tejun Heo
2018-01-10 18:14 ` Tejun Heo
2018-01-10 18:45 ` Steven Rostedt
2018-01-10 18:45 ` Steven Rostedt
2018-01-10 18:41 ` Steven Rostedt
2018-01-10 18:41 ` Steven Rostedt
2018-01-10 18:57 ` Tejun Heo
2018-01-10 18:57 ` Tejun Heo
2018-01-10 19:17 ` Steven Rostedt
2018-01-10 19:17 ` Steven Rostedt
2018-01-10 19:34 ` Tejun Heo
2018-01-10 19:34 ` Tejun Heo
2018-01-10 19:44 ` Steven Rostedt
2018-01-10 19:44 ` Steven Rostedt
2018-01-10 22:44 ` Tejun Heo
2018-01-10 22:44 ` Tejun Heo
2018-01-11 5:35 ` Sergey Senozhatsky
2018-01-11 5:35 ` Sergey Senozhatsky
2018-01-11 4:58 ` Sergey Senozhatsky
2018-01-11 4:58 ` Sergey Senozhatsky
2018-01-11 9:34 ` Petr Mladek
2018-01-11 9:34 ` Petr Mladek
2018-01-11 10:38 ` Sergey Senozhatsky
2018-01-11 10:38 ` Sergey Senozhatsky
2018-01-11 11:50 ` Petr Mladek
2018-01-11 11:50 ` Petr Mladek
2018-01-11 16:29 ` Steven Rostedt
2018-01-11 16:29 ` Steven Rostedt
2018-01-12 1:30 ` Steven Rostedt
2018-01-12 1:30 ` Steven Rostedt
2018-01-12 2:55 ` Steven Rostedt
2018-01-12 2:55 ` Steven Rostedt
2018-01-12 4:20 ` Steven Rostedt
2018-01-12 4:20 ` Steven Rostedt
2018-01-16 19:44 ` Tejun Heo
2018-01-16 19:44 ` Tejun Heo
2018-01-17 9:12 ` Petr Mladek
2018-01-17 9:12 ` Petr Mladek
2018-01-17 15:15 ` Tejun Heo
2018-01-17 15:15 ` Tejun Heo
2018-01-17 17:12 ` Steven Rostedt
2018-01-17 17:12 ` Steven Rostedt
2018-01-17 18:42 ` Steven Rostedt
2018-01-17 18:42 ` Steven Rostedt
2018-01-19 18:20 ` Steven Rostedt
2018-01-19 18:20 ` Steven Rostedt
2018-01-20 7:14 ` Sergey Senozhatsky
2018-01-20 7:14 ` Sergey Senozhatsky
2018-01-20 15:49 ` Steven Rostedt
2018-01-20 15:49 ` Steven Rostedt
2018-01-21 14:15 ` Sergey Senozhatsky
2018-01-21 14:15 ` Sergey Senozhatsky
2018-01-21 21:04 ` Steven Rostedt
2018-01-21 21:04 ` Steven Rostedt
2018-01-22 8:56 ` Sergey Senozhatsky
2018-01-22 8:56 ` Sergey Senozhatsky
2018-01-22 10:28 ` Sergey Senozhatsky
2018-01-22 10:28 ` Sergey Senozhatsky
2018-01-22 10:36 ` Sergey Senozhatsky
2018-01-22 10:36 ` Sergey Senozhatsky
2018-01-23 6:40 ` Sergey Senozhatsky
2018-01-23 6:40 ` Sergey Senozhatsky
2018-01-23 7:05 ` Sergey Senozhatsky
2018-01-23 7:05 ` Sergey Senozhatsky
2018-01-23 7:31 ` Sergey Senozhatsky
2018-01-23 7:31 ` Sergey Senozhatsky
2018-01-23 14:56 ` Steven Rostedt
2018-01-23 14:56 ` Steven Rostedt
2018-01-23 15:21 ` Sergey Senozhatsky
2018-01-23 15:21 ` Sergey Senozhatsky
2018-01-23 15:41 ` Steven Rostedt
2018-01-23 15:41 ` Steven Rostedt
2018-01-23 15:43 ` Tejun Heo
2018-01-23 15:43 ` Tejun Heo
2018-01-23 16:12 ` Sergey Senozhatsky
2018-01-23 16:12 ` Sergey Senozhatsky
2018-01-23 16:13 ` Steven Rostedt
2018-01-23 16:13 ` Steven Rostedt
2018-01-23 17:21 ` Tejun Heo
2018-01-23 17:21 ` Tejun Heo
2018-04-23 5:35 ` Sergey Senozhatsky
2018-04-23 5:35 ` Sergey Senozhatsky
2018-01-23 16:01 ` Sergey Senozhatsky
2018-01-23 16:01 ` Sergey Senozhatsky
2018-01-23 16:24 ` Steven Rostedt
2018-01-23 16:24 ` Steven Rostedt
2018-01-24 2:11 ` Sergey Senozhatsky
2018-01-24 2:11 ` Sergey Senozhatsky
2018-01-24 2:52 ` Steven Rostedt
2018-01-24 2:52 ` Steven Rostedt
2018-01-24 4:44 ` Sergey Senozhatsky
2018-01-24 4:44 ` Sergey Senozhatsky
2018-01-23 17:22 ` Tejun Heo
2018-01-23 17:22 ` Tejun Heo
2018-01-20 12:19 ` Tejun Heo
2018-01-20 12:19 ` Tejun Heo
2018-01-20 14:51 ` Steven Rostedt
2018-01-20 14:51 ` Steven Rostedt
2018-01-17 20:05 ` Tejun Heo
2018-01-17 20:05 ` Tejun Heo
2018-01-18 5:43 ` Sergey Senozhatsky
2018-01-18 5:43 ` Sergey Senozhatsky
2018-01-18 11:51 ` Petr Mladek
2018-01-18 11:51 ` Petr Mladek
2018-01-18 5:42 ` Sergey Senozhatsky
2018-01-18 5:42 ` Sergey Senozhatsky
2018-01-12 3:12 ` Sergey Senozhatsky
2018-01-12 3:12 ` Sergey Senozhatsky
2018-01-12 2:56 ` Sergey Senozhatsky
2018-01-12 2:56 ` Sergey Senozhatsky
2018-01-12 3:21 ` Steven Rostedt
2018-01-12 3:21 ` Steven Rostedt
2018-01-12 10:05 ` Sergey Senozhatsky
2018-01-12 10:05 ` Sergey Senozhatsky
2018-01-12 12:21 ` Steven Rostedt
2018-01-12 12:21 ` Steven Rostedt
2018-01-12 12:55 ` Petr Mladek
2018-01-12 12:55 ` Petr Mladek
2018-01-13 7:31 ` Sergey Senozhatsky
2018-01-13 7:31 ` Sergey Senozhatsky
2018-01-15 8:51 ` Petr Mladek
2018-01-15 8:51 ` Petr Mladek
2018-01-15 9:48 ` Sergey Senozhatsky
2018-01-15 9:48 ` Sergey Senozhatsky
2018-01-16 5:16 ` Sergey Senozhatsky
2018-01-16 5:16 ` Sergey Senozhatsky
2018-01-16 9:08 ` Petr Mladek
2018-01-16 9:08 ` Petr Mladek
2018-01-15 12:08 ` Steven Rostedt
2018-01-15 12:08 ` Steven Rostedt
2018-01-16 4:51 ` Sergey Senozhatsky
2018-01-16 4:51 ` Sergey Senozhatsky
2018-01-13 7:28 ` Sergey Senozhatsky
2018-01-13 7:28 ` Sergey Senozhatsky
2018-01-15 10:17 ` Petr Mladek
2018-01-15 10:17 ` Petr Mladek
2018-01-15 11:50 ` Petr Mladek
2018-01-15 11:50 ` Petr Mladek
2018-01-16 6:10 ` Sergey Senozhatsky
2018-01-16 6:10 ` Sergey Senozhatsky
2018-01-16 9:36 ` Petr Mladek
2018-01-16 9:36 ` Petr Mladek
2018-01-16 10:10 ` Sergey Senozhatsky
2018-01-16 10:10 ` Sergey Senozhatsky
2018-01-16 16:06 ` Steven Rostedt
2018-01-16 16:06 ` Steven Rostedt
2018-01-16 5:23 ` Sergey Senozhatsky
2018-01-16 5:23 ` Sergey Senozhatsky
2018-01-15 12:06 ` Steven Rostedt
2018-01-15 12:06 ` Steven Rostedt
2018-01-15 14:45 ` Petr Mladek
2018-01-15 14:45 ` Petr Mladek
2018-01-16 2:23 ` Sergey Senozhatsky
2018-01-16 2:23 ` Sergey Senozhatsky
2018-01-16 4:47 ` Sergey Senozhatsky
2018-01-16 4:47 ` Sergey Senozhatsky
2018-01-16 10:19 ` Petr Mladek
2018-01-16 10:19 ` Petr Mladek
2018-01-17 2:24 ` Sergey Senozhatsky
2018-01-17 2:24 ` Sergey Senozhatsky
2018-01-16 15:45 ` Steven Rostedt
2018-01-16 15:45 ` Steven Rostedt
2018-01-17 2:18 ` Sergey Senozhatsky
2018-01-17 2:18 ` Sergey Senozhatsky
2018-01-17 13:04 ` Petr Mladek
2018-01-17 13:04 ` Petr Mladek
2018-01-17 15:24 ` Steven Rostedt
2018-01-17 15:24 ` Steven Rostedt
2018-01-18 4:31 ` Sergey Senozhatsky
2018-01-18 4:31 ` Sergey Senozhatsky
2018-01-18 15:22 ` Steven Rostedt
2018-01-18 15:22 ` Steven Rostedt
2018-01-16 10:13 ` Petr Mladek
2018-01-16 10:13 ` Petr Mladek
2018-01-17 6:29 ` Sergey Senozhatsky
2018-01-17 6:29 ` Sergey Senozhatsky
2018-01-16 1:46 ` Sergey Senozhatsky
2018-01-16 1:46 ` Sergey Senozhatsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180111051524.GC494@jagdpanzerIV \
--to=sergey.senozhatsky.work@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=byungchul.park@lge.com \
--cc=dave.hansen@intel.com \
--cc=hannes@cmpxchg.org \
--cc=jack@suse.cz \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=mgorman@suse.de \
--cc=mhocko@kernel.org \
--cc=pavel@ucw.cz \
--cc=penguin-kernel@I-love.SAKURA.ne.jp \
--cc=peterz@infradead.org \
--cc=pmladek@suse.com \
--cc=rostedt@goodmis.org \
--cc=rostedt@home.goodmis.org \
--cc=sergey.senozhatsky@gmail.com \
--cc=tj@kernel.org \
--cc=torvalds@linux-foundation.org \
--cc=vbabka@suse.cz \
--cc=xiyou.wangcong@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.