From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754333AbbLJPIX (ORCPT ); Thu, 10 Dec 2015 10:08:23 -0500 Received: from mail-pf0-f170.google.com ([209.85.192.170]:36605 "EHLO mail-pf0-f170.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753882AbbLJPIV (ORCPT ); Thu, 10 Dec 2015 10:08:21 -0500 Date: Fri, 11 Dec 2015 00:06:48 +0900 From: Sergey Senozhatsky To: Jan Kara Cc: Andrew Morton , Petr Mladek , KY Srinivasan , Steven Rostedt , linux-kernel@vger.kernel.org, Sergey Senozhatsky , Sergey Senozhatsky Subject: Re: [PATCH 2/7] printk: Start printing handover kthreads on demand Message-ID: <20151210150648.GC540@swordfish> References: <20151210145607.GB540@swordfish> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151210145607.GB540@swordfish> User-Agent: Mutt/1.5.24 (2015-08-30) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On (12/10/15 23:56), Sergey Senozhatsky wrote: > > A silly minor nitpick > ah.. nope, I lied. this part >+static int printk_start_offload_kthreads(void) > { >- struct console *con; > int i; > struct task_struct *task; > >+ /* Does handover of printing make any sense? */ >+ if (printk_offload_chars == 0 || num_possible_cpus() <= 1) >+ return 0; >+ for (i = 0; i < PRINTING_TASKS; i++) { >+ if (printing_kthread[i]) >+ continue; >+ task = kthread_run(printing_task, NULL, "print/%d", i); >+ if (IS_ERR(task)) >+ goto out_err; >+ printing_kthread[i] = task; >+ } >+ return 0; >+out_err: >+ pr_err("printk: Cannot create printing thread: %ld\n", PTR_ERR(task)); >+ /* Disable offloading if creating kthreads failed */ >+ printk_offload_chars = 0; >+ return PTR_ERR(task); >+} we can keep printk_offload_chars if we have created at least one thread, in theory. if we are in heavy or nearly hevy (which is 'a process will spin in unlock_console() with preemption_disabled for far too long') printk traffic case (and that's what I want to fix from my side) then we have at least 1 process doing printk->console_unlock() very often, so breaking that 'while (1)' console_unlock() loop is a good thing here. which may be a corner case. if we want to zero `printk_offload_chars' then how about bringing back Tejun Heo's "printk: do cond_resched() between lines while outputting to consoles"? Besides, we need this for !CONFIG_PRINTK_OFFLOAD kernel. ===8<==== if printk_start_offload_kthreads() has failed or if for some other reason `printk_offload_chars' ended up to be 0, we still need to try to break up long console_unlock() loops and try to reschedule. Signed-off-by: Sergey Senozhatsky --- kernel/printk/printk.c | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c index 21b0fb9..fc8c493 100644 --- a/kernel/printk/printk.c +++ b/kernel/printk/printk.c @@ -2285,6 +2285,16 @@ static bool cpu_stop_printing(int printed_chars) return false; } +static bool cpu_should_cond_resched(bool do_cond_resched) +{ + /* Oops? Print everything now to maximize chances user will see it */ + if (oops_in_progress) + return false; + if (!printk_offload_chars && do_cond_resched) + return true; + return false; +} + /** * console_unlock - unlock the console system * @@ -2309,7 +2319,7 @@ void console_unlock(void) unsigned long flags; bool wake_klogd = false; bool retry; - bool hand_over = false; + bool hand_over = false, do_cond_resched; int printed_chars = 0; if (console_suspended) { @@ -2317,6 +2327,15 @@ void console_unlock(void) return; } + /* + * We may end up dumping a lot of lines, for example, if called + * from console registration path, and should invoke cond_resched() + * between lines if allowable. Not doing so can cause a very long + * scheduling stall on a slow console leading to RCU stall and + * softlockup warnings which exacerbate the issue with more + * messages practically incapacitating the system. + */ + do_cond_resched = console_may_schedule; console_may_schedule = 0; /* flush buffered message fragment immediately to console */ @@ -2397,6 +2416,12 @@ skip: call_console_drivers(level, ext_text, ext_len, text, len); start_critical_timings(); printed_chars += len; + + if (unlikely(cpu_should_cond_resched(do_cond_resched))) { + raw_spin_unlock_irqrestore(&print_lock, flags); + cond_resched(); + raw_spin_lock_irqsave(&print_lock, flags); + } } /* Release the exclusive_console once it is used */ -- 2.6.3