From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757478AbbLCBKa (ORCPT ); Wed, 2 Dec 2015 20:10:30 -0500 Received: from mail-pa0-f48.google.com ([209.85.220.48]:35637 "EHLO mail-pa0-f48.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756924AbbLCBK2 (ORCPT ); Wed, 2 Dec 2015 20:10:28 -0500 Date: Thu, 3 Dec 2015 10:11:29 +0900 From: Sergey Senozhatsky To: tj@kernel.org Cc: akpm@linux-foundation.org, calvinowens@fb.com, davej@codemonkey.org.uk, jack@suse.com, kyle@kernel.org, stable@vger.kernel.org, mm-commits@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: + printk-do-cond_resched-between-lines-while-outputting-to-consoles.patch added to -mm tree Message-ID: <20151203011129.GA510@swordfish> References: <565f855a./bN6NB3bZKjpF4Wa%akpm@linux-foundation.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <565f855a./bN6NB3bZKjpF4Wa%akpm@linux-foundation.org> User-Agent: Mutt/1.5.24 (2015-08-30) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hello, On (12/02/15 15:57), akpm@linux-foundation.org wrote: [..] > @console_may_schedule tracks whether console_sem was acquired through lock > or trylock. If the former, we're inside a sleepable context and > console_conditional_schedule() performs cond_resched(). This allows > console drivers which use console_lock for synchronization to yield while > performing time-consuming operations such as scrolling. > > However, the actual console outputting is performed while holding irq-safe > logbuf_lock, so console_unlock() clears @console_may_schedule before > starting outputting lines. Also, only a few drivers call > console_conditional_schedule() to begin with. This means that when a lot > of lines need to be output by console_unlock(), for example on a console > registration, the task doing console_unlock() may not yield for a long > time on a non-preemptible kernel. > > If this happens with a slow console devices, for example a serial console, > the outputting task may occupy the cpu for a very long time. Long enough > to trigger softlockup and/or RCU stall warnings, which in turn pile more > messages, sometimes enough to trigger the next cycle of warnings > incapacitating the system. > > Fix it by making console_unlock() insert cond_resched() between lines if > @console_may_schedule. CPU2 still can cause lots of troubles. consider CPU0 CPU1 CPU2 printk ... printk_deferred printk wake_up_klogd wake_up_klogd_work_func console_trylock console_unlock printk_deferred() may be issued by scheduler, for example. -ss > Signed-off-by: Tejun Heo > Reported-by: Calvin Owens > Acked-by: Jan Kara > Cc: Dave Jones > Cc: Kyle McMartin > Cc: > Signed-off-by: Andrew Morton > --- > > kernel/printk/printk.c | 16 +++++++++++++++- > 1 file changed, 15 insertions(+), 1 deletion(-) > > diff -puN kernel/printk/printk.c~printk-do-cond_resched-between-lines-while-outputting-to-consoles kernel/printk/printk.c > --- a/kernel/printk/printk.c~printk-do-cond_resched-between-lines-while-outputting-to-consoles > +++ a/kernel/printk/printk.c > @@ -2234,13 +2234,24 @@ void console_unlock(void) > static u64 seen_seq; > unsigned long flags; > bool wake_klogd = false; > - bool retry; > + bool do_cond_resched, retry; > > if (console_suspended) { > up_console_sem(); > return; > } > > + /* > + * Console drivers are called under logbuf_lock, so > + * @console_may_schedule should be cleared before; however, we may > + * end up dumping a lot of lines, for example, if called from > + * console registration path, and should invoke cond_resched() > + * between lines if allowable. Not doing so can cause a very long > + * scheduling stall on a slow console leading to RCU stall and > + * softlockup warnings which exacerbate the issue with more > + * messages practically incapacitating the system. > + */ > + do_cond_resched = console_may_schedule; > console_may_schedule = 0; > > /* flush buffered message fragment immediately to console */ > @@ -2312,6 +2323,9 @@ skip: > call_console_drivers(level, ext_text, ext_len, text, len); > start_critical_timings(); > local_irq_restore(flags); > + > + if (do_cond_resched) > + cond_resched(); > } > console_locked = 0; > > _ > > Patches currently in -mm which might be from tj@kernel.org are > > printk-do-cond_resched-between-lines-while-outputting-to-consoles.patch > > -- > To unsubscribe from this list: send the line "unsubscribe mm-commits" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html >