From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7CF7C433F5 for ; Wed, 30 Mar 2022 12:10:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344963AbiC3MLn (ORCPT ); Wed, 30 Mar 2022 08:11:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58166 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345317AbiC3LyW (ORCPT ); Wed, 30 Mar 2022 07:54:22 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B4623278543; Wed, 30 Mar 2022 04:50:53 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 43937B81C36; Wed, 30 Mar 2022 11:50:53 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 35E30C36AE5; Wed, 30 Mar 2022 11:50:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1648641051; bh=W28dUiUQFRtYyYKOqfOF5acJxI2YPCJwPhAGm1Nfn6A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WwKTBb1msXbuppPLM9IMgR3lgtWlC2y7P+MgQr/AecaR/t1iJLMZ2jNKBvrKtpSxr ysK4mdvCmRRW4IOXgtl5KF4qmDRg3ZXYjm3aie4605ioNEtTB+ey87cqIJylLQwkzl cGGhr2jOWYvifnwIKbMp/Q8SpclZ/VtcvGPkMwfvqbhKh8Q8t9hGGAkmQhuI1pyt64 wPFwksLFsu9culfmffa4GbTVFP2qhwX/CbplmfDJ4CYXnBlt61bxBRNS9YafdoouE3 YlpDOCbcLUgHJwYS6VjJ48E/mRzuFpHnr0+LESOmi5XgIlOl9QhiFf62YDzj3t3Ro7 Uw6y8Za7PZ13g== From: Sasha Levin To: linux-kernel@vger.kernel.org, stable@vger.kernel.org Cc: Stephen Brennan , Petr Mladek , Sergey Senozhatsky , Sasha Levin , gregkh@linuxfoundation.org Subject: [PATCH AUTOSEL 5.15 30/50] printk: Avoid livelock with heavy printk during panic Date: Wed, 30 Mar 2022 07:49:44 -0400 Message-Id: <20220330115005.1671090-30-sashal@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220330115005.1671090-1-sashal@kernel.org> References: <20220330115005.1671090-1-sashal@kernel.org> MIME-Version: 1.0 X-stable: review X-Patchwork-Hint: Ignore Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Stephen Brennan [ Upstream commit 13fb0f74d7029df3b8137f11ef955e578a4a4a60 ] During panic(), if another CPU is writing heavily the kernel log (e.g. via /dev/kmsg), then the panic CPU may livelock writing out its messages to the console. Note when too many messages are dropped during panic and suppress further printk, except from the panic CPU. This could result in some important messages being dropped. However, messages are already being dropped, so this approach at least prevents a livelock. Reviewed-by: Petr Mladek Signed-off-by: Stephen Brennan Reviewed-by: Sergey Senozhatsky Signed-off-by: Petr Mladek Link: https://lore.kernel.org/r/20220202171821.179394-4-stephen.s.brennan@oracle.com Signed-off-by: Sasha Levin --- kernel/printk/printk.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c index f9aebe88cc53..f37889b6cf9f 100644 --- a/kernel/printk/printk.c +++ b/kernel/printk/printk.c @@ -93,6 +93,12 @@ EXPORT_SYMBOL_GPL(console_drivers); */ int __read_mostly suppress_printk; +/* + * During panic, heavy printk by other CPUs can delay the + * panic and risk deadlock on console resources. + */ +int __read_mostly suppress_panic_printk; + #ifdef CONFIG_LOCKDEP static struct lockdep_map console_lock_dep_map = { .name = "console_lock" @@ -2222,6 +2228,10 @@ asmlinkage int vprintk_emit(int facility, int level, if (unlikely(suppress_printk)) return 0; + if (unlikely(suppress_panic_printk) && + atomic_read(&panic_cpu) != raw_smp_processor_id()) + return 0; + if (level == LOGLEVEL_SCHED) { level = LOGLEVEL_DEFAULT; in_sched = true; @@ -2607,6 +2617,7 @@ void console_unlock(void) { static char ext_text[CONSOLE_EXT_LOG_MAX]; static char text[CONSOLE_LOG_MAX]; + static int panic_console_dropped; unsigned long flags; bool do_cond_resched, retry; struct printk_info info; @@ -2661,6 +2672,10 @@ void console_unlock(void) if (console_seq != r.info->seq) { console_dropped += r.info->seq - console_seq; console_seq = r.info->seq; + if (panic_in_progress() && panic_console_dropped++ > 10) { + suppress_panic_printk = 1; + pr_warn_once("Too many dropped messages. Suppress messages on non-panic CPUs to prevent livelock.\n"); + } } if (suppress_message_printing(r.info->level)) { -- 2.34.1