From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from CWXP265CU010.outbound.protection.outlook.com (mail-ukwestazon11022091.outbound.protection.outlook.com [52.101.101.91]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7F1AF23BD1B for ; Tue, 4 Aug 2026 00:16:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.101.91 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785802590; cv=fail; b=s1qbzu/uNRGvCGVZnBAEgMKWn+DbvSQIfm9Fzzy8WbBBYxLufrGxqzcHwgoxRwsBlblJRvjBOwyIZuDCV677fKEb4AceQ4hzHO4DuFTKvYbqYNWuqdryW1oORfqGotOftsjsmf9SnR7KUizPkw6IFK+ixDy6XXcY2sqVoeNqdEs= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785802590; c=relaxed/simple; bh=gExQT7IKyyvmfOhx3zvJTjsF9CDxv3HjFO1KzSnoSOY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=nYZuoEwWStYbz/9BawSe2ZY+3GEIQAv1V1zA9vIFQ99J/1LHx92rCzlPe1E8RA7tBXMkW6T51SEtUJR/QrKXHtXUF+9oKARihec9ccVSGeCGXXHvCR5fK4zwfJ7KGdGsqG5G4J+ukeDSCsanWbQWVET/f6p20EzUR7jzRRXptr0= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=atomlin.com; spf=pass smtp.mailfrom=atomlin.com; arc=fail smtp.client-ip=52.101.101.91 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=atomlin.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=atomlin.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=YrK12qQbXdHTXrRwUU/MY8cbVoFaaG5wMHjoWo2vQOdjh/4xy4UPC7bHnFHj1gVBk0BolqK0qhEqaDTKW9GTpiynbhu50fuFhdKZoYaoR83CD8m03oSBV/0umJgI25qm2Qr4O1XYUKAOL+gej1Iq139/SrGUMZjAOtB7xAe/rSDv18vjoRpDl5lU5g12euvKNZ1M0a0dM08Vb+W34/6SVWkFAOkd2Ln6dKrXx5at5eAHumgdjkHOZNJOXPsEfDGmUmoWo/AcUjChrutm/vwP3jCRW6bXoXy6q5B3RzznZ4TbD4GyqNPm0WTtpEe4Qrde4pGvKGnxa67jkv4yHZLCfg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:MIME-Version; bh=9jG6kD+ElTfAJMz5CmFmcUw6LxRVCbo1iE9BpeB4XvE=; b=eolB/W+2POEzbYWmj0yMMi5E2X0/YCmdb/54Wx80iJHhk3FOt4Lic/CX+6bdw9EGHvkpIRfLKBfz8B0wUwUFufWnKT81MF9jP5B6N4Ij1HU0i2y9Eo7Ukix9QSsVw5HEzgiTcTP1iGXyqHkgco1vMBjP/oRdumYwtXmiV2l9HAKM7FlxFFfY1Fh/oRlSTMDqo6FG107ecTqD8mMuv3sd2s4E2C7Xd3NCF8n1YfCD5arZyQKOfA0/2JwqGs+8d3+8BI8XFdBYy0MAKk4388hIhB+HLtCgtn/u3eLZH49RA5oZelkEvainNy5gtEOYdaqEpHEwutxkfHGkHss39msSNw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=atomlin.com; dmarc=pass action=none header.from=atomlin.com; dkim=pass header.d=atomlin.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=atomlin.com; Received: from LO6P123MB6616.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:2b2::11) by LO2P123MB3741.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:12e::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.270.18; Tue, 4 Aug 2026 00:16:25 +0000 Received: from LO6P123MB6616.GBRP123.PROD.OUTLOOK.COM ([fe80::3975:c5e7:bc1a:f383]) by LO6P123MB6616.GBRP123.PROD.OUTLOOK.COM ([fe80::3975:c5e7:bc1a:f383%4]) with mapi id 15.21.0270.017; Tue, 4 Aug 2026 00:16:25 +0000 From: Aaron Tomlin To: akpm@linux-foundation.org, lance.yang@linux.dev, mhiramat@kernel.org, pmladek@suse.com Cc: linux-kernel@vger.kernel.org, david.laight.linux@gmail.com, atomlin@atomlin.com, neelx@suse.com, sean@ashe.io, chjohnst@gmail.com, steve@abita.co, mproche@gmail.com, nick.lange@gmail.com Subject: [PATCH 1/2] hung_task: Reset warning budget when problem gets resolved Date: Mon, 3 Aug 2026 20:15:41 -0400 Message-ID: <20260804001542.249639-2-atomlin@atomlin.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260804001542.249639-1-atomlin@atomlin.com> References: <20260804001542.249639-1-atomlin@atomlin.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: BN9PR03CA0396.namprd03.prod.outlook.com (2603:10b6:408:111::11) To CWLP123MB6607.GBRP123.PROD.OUTLOOK.COM (2603:10a6:400:183::5) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LO6P123MB6616:EE_|LO2P123MB3741:EE_ X-MS-Office365-Filtering-Correlation-Id: b6fd32ca-68a0-4727-f76d-08def1bd9eb1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|366016|7416014|376014|1800799024|56012099006|10067099003|18002099003|22082099003; X-Microsoft-Antispam-Message-Info: CQT9mmrPTqYWeSELFOdZBlAm1kEGfJkPBGhRBXKHSnEeSVC8rzH4ZsLdNnYnCWWUyiuwrpQMhb5ZprSvyea/cW0mwj1iFYxjlQDaITcEFWUzBlB32v7sfq5s37frFPzD9Zm7OHGRvNm7Ehqun2bx+8z68ybtvwRD+9ZnkRi7wBgSWMe7X8q1bK9jDv3S9SZmTkB89nDp5U0FSEMhDJleikXdb5MQDdLE06rjm83k6tqPf6IPLaJ+9zbtmCU7XxQY8D0DlfeMf6/JwaJiSIHA069gxgdiKj9dLEaz6CsmxUyogNKWxwA6vsj/v3PHjWde5FrXCzWv2F+rNq0Hr+soYs0WDvpfp2VG6dkaIDmC2+zKvtGWwtDBuE7N1YYT/3Hh73oqyjMcVqcLaw/OeQrgbmBM6aOwOc13lQF4IK4ukftSXWp0yO01QjYjOvbscNZqVDdBDCRj9Hc3DeOuY8a83ne95mCBhFC9Vta/k/JgadMZqmPdVpOPRSAxUAm/RPL4BpgYLwmzJxxyYab/xKbMXXddioYagEyBE0H+HJsLMJkE/G2+8YrO+s3YDNy7GedyMVbtt5cECqW7R38h6jIWdpQX1wNJc7AVcys7TdOon+yAunboJ1oYJ36rKD6wF7JbZ3MvgSRKljboQwH1SgEjXiLb94kGuJ7XL+y+9Xsw4x0= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LO6P123MB6616.GBRP123.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(366016)(7416014)(376014)(1800799024)(56012099006)(10067099003)(18002099003)(22082099003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?vyxmSuoWcJZFzcG3fCQ63WwjIquT4+1CDVY36IDIGntqGlzIBmDjfQqYCTaC?= =?us-ascii?Q?Q2k85EflvgJAZ0s1t9fsadKGzmOYr/axS5N4sj9nwCB9MPOyEPaAyFQyvKTA?= =?us-ascii?Q?+iXF0zx9ZP/ZdXbzNl3Cr3J/foXZVliS16hr+VZNgadOTKVhKOV8CqXDXlk4?= =?us-ascii?Q?/QNPBKk8BO+/Kcy3DLLVkO7b1zTOA6F8+81lUgBiABh3tIue9b/YU3AODabs?= =?us-ascii?Q?ZTl1aqwsqXXVj1fihV1RZDhSYQOsaIdy8NaCQ6lyeNGPbM6/rtAbdIJRPaKP?= =?us-ascii?Q?oTptReEoSt860XYTIc4SxAEbCPlwJO3jGsiqYmTDQFRohR5OjMBUY80YC2Zl?= =?us-ascii?Q?783TUsfl1rX1mr4lG9ghyeCHwmvMPcudLBWq6T6ndUfgELEKUlubFrQKJ82r?= =?us-ascii?Q?cMLAebod+FY+A472NZvSuBrijliQ3SD6beYBJ3cbDlvxQNldew5u8JxZ97Bg?= =?us-ascii?Q?vRUWNI/zGZyli3hG5/QO8lsuPhu9n1Uy7EK+JisvmrxSHOZs9+y7MqaYD5Ij?= =?us-ascii?Q?MqctxzFNIiGvSPN3Fianq0x32fR6cHfRZc8YdrJG+h40GH+PjLBP3ugR6yeB?= =?us-ascii?Q?K65SJwj4+nd4IiV2R4PNty+eo5NK5wtl2vOlTD4plDqRx3L9USOcHJtw929Z?= =?us-ascii?Q?CmIHcNDFG/vjdzPXZ/nUSeAXJ/ulfjXs2aoIj8n9cFA/Y7H4eGiaq3qx5vow?= =?us-ascii?Q?boZ3Zn2+BhRHhg8X18CxqMVGuQLFuZqAMMUKM4dxIItpcjgehO+n1Bs5nNqa?= =?us-ascii?Q?8R3vwQJEtPGd+8tjqQvSOXsdHN56m/HS0/bThHuFOh9rrNR1WmeXON7mddZ+?= =?us-ascii?Q?CinIMdFlrcb6IRSlIp9NqqojR73i23FnEPn6gtEjF3l+TvsVir0ChromwlOd?= =?us-ascii?Q?ydhCReW4kYECm5igWjPWACBzPDVmQSZ6tuTA9aTpmdyEOmpSSPTr2hrYw9mz?= =?us-ascii?Q?eVxtF6oHdYMXu7m7A6ibIThjnFm4MR3Wb927+f1f0kLk3FbwmUETneEISQxQ?= =?us-ascii?Q?ni3L0FiSEUh4IVSKRvFGGdprG0wz971mWM4lx1I2hxn6sM3WaPjcsvZnKmhl?= =?us-ascii?Q?QXFDNXKq35acPSLAs0Za4MHZuNqPyYHTHkwfvBPe4WxReWh0DGlTpNQiCGwB?= =?us-ascii?Q?cRd4Bfd3rtFM0IHArWlEyDnx27BlcjtLu0sJCWCAy+UQXZXLMimIVCEEcfa2?= =?us-ascii?Q?R7I6SLGTrsAU2orPMETnm1jpRsZlsjs1bso5QcBJCKEhMSovJpkyYSjmTe9y?= =?us-ascii?Q?CFDZQhgAoT7BgFZDIzPA9TtxPN7Qg6RzODf1tsYoeKhlue3iO4Ovcd8aaKdW?= =?us-ascii?Q?2ZNaSAZKzsmVPokVQ/cyQPZuFG2NZjbeY2iFnv3tJJszMpnWU4XN/p+sDYEr?= =?us-ascii?Q?3dViRM0zIKa/vjIdKL3J9Tiu0pndEkncfpdhlioDzXij0pojv+p9mS/ZldOQ?= =?us-ascii?Q?GPBy1CynL7x2nERQ9028Y+0Xq1dEPgRJvpzatsTu4dUxF6qrAKzaQvG6SoIl?= =?us-ascii?Q?jBYFLMFw/eqUS0LK+9fvj4fQ7vwv3ls6QoPB76Xu2hG3fo9Nh6nqk2ZiLKzo?= =?us-ascii?Q?X2gI7CLbD8wOkOZenmC1zdprf8QK3sdZIC/fGfRYi3twnhUYqgBIyir87WTW?= =?us-ascii?Q?IW2rISmOAv7jTpldcOIJVpBPFiodMgcawbd8LxEgnWk7Skd6vcKZD71g0E6V?= =?us-ascii?Q?hB59KIazjeu8mwGzqm2eGlSNDpI6OUuWv7B0f1cYWqHhGGBgD84I/cJO57l0?= =?us-ascii?Q?euzTGUV+eA=3D=3D?= X-OriginatorOrg: atomlin.com X-MS-Exchange-CrossTenant-Network-Message-Id: b6fd32ca-68a0-4727-f76d-08def1bd9eb1 X-MS-Exchange-CrossTenant-AuthSource: CWLP123MB6607.GBRP123.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Aug 2026 00:16:25.0642 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e6a32402-7d7b-4830-9a2b-76945bbbcb57 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: yPg1aMITwqTu30EiK43wQrvmiD6seyA+frRfTaHatSZ6VJDtWn/mdGhZCSLNrf7EevQnPuvyKCJLz0MoZPUwLQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LO2P123MB3741 From: Petr Mladek sysctl_hung_task_warnings counts how many hung tasks are reported. The watchdog does not report anything once the limit is reached. Currently, this budget is decremented permanently, meaning the kernel is left blind to subsequent hung tasks even after the original issue resolves. Keep the global sysctl_hung_task_warnings intact, and instead decrement a copy (hung_task_warnings_printed) when warnings are printed. Reset the copy back to the configured sysctl_hung_task_warnings limit once the problem on the system gets resolved and check_hung_uninterruptible_tasks() detects no hung tasks in a check interval. Also keep the copy updated when the global sysctl_hung_task_warnings value is updated via sysctl, and update documentation to reflect the new behavior. Assisted-by: gemini-3.5-flash Signed-off-by: Petr Mladek --- Documentation/admin-guide/sysctl/kernel.rst | 6 ++-- kernel/hung_task.c | 34 ++++++++++++++++----- 2 files changed, 31 insertions(+), 9 deletions(-) diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst index c6994e55d141..eaa1d837c739 100644 --- a/Documentation/admin-guide/sysctl/kernel.rst +++ b/Documentation/admin-guide/sysctl/kernel.rst @@ -459,8 +459,10 @@ hung_task_warnings ================== The maximum number of warnings to report. During a check interval -if a hung task is detected, this value is decreased by 1. -When this value reaches 0, no more warnings will be reported. +if a hung task is detected, the internal warning budget is decreased by 1. +When this budget reaches 0, no more warnings will be reported. +The warning budget is reset back to the configured limit when the problem on the +system gets resolved and no hung task is found. This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled. -1: report an infinite number of warnings. diff --git a/kernel/hung_task.c b/kernel/hung_task.c index 6fcc94ce4ca9..4e1fb0db79d1 100644 --- a/kernel/hung_task.c +++ b/kernel/hung_task.c @@ -59,6 +59,8 @@ static unsigned long __read_mostly sysctl_hung_task_check_interval_secs; static int __read_mostly sysctl_hung_task_warnings = 10; +static int hung_task_warnings_printed = 10; + static int __read_mostly did_panic; static bool hung_task_call_panic; @@ -247,9 +249,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout, * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain * accordingly */ - if (sysctl_hung_task_warnings || hung_task_call_panic) { - if (sysctl_hung_task_warnings > 0) - sysctl_hung_task_warnings--; + if (hung_task_warnings_printed || hung_task_call_panic) { + if (hung_task_warnings_printed > 0) + hung_task_warnings_printed--; pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n", t->comm, t->pid, t->in_iowait ? " in I/O wait" : "", (jiffies - t->last_switch_time) / HZ); @@ -264,7 +266,7 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout, sched_show_task(t); debug_show_blocker(t, timeout); - if (!sysctl_hung_task_warnings) + if (!hung_task_warnings_printed) pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n"); } @@ -304,7 +306,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) unsigned long last_break = jiffies; struct task_struct *g, *t; unsigned long this_round_count; - int need_warning = sysctl_hung_task_warnings; + int need_warning = hung_task_warnings_printed; unsigned long si_mask = hung_task_si_mask; /* @@ -340,8 +342,10 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) unlock: rcu_read_unlock(); - if (!this_round_count) + if (!this_round_count) { + hung_task_warnings_printed = sysctl_hung_task_warnings; return; + } if (need_warning || hung_task_call_panic) { si_mask |= SYS_INFO_LOCKS; @@ -425,6 +429,22 @@ static int proc_dohung_task_timeout_secs(const struct ctl_table *table, int writ return ret; } +static int proc_dohung_task_warnings(const struct ctl_table *table, int write, + void *buffer, + size_t *lenp, loff_t *ppos) +{ + int ret; + + ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos); + + if (ret || !write) + return ret; + + hung_task_warnings_printed = sysctl_hung_task_warnings; + + return 0; +} + /* * This is needed for proc_doulongvec_minmax of sysctl_hung_task_timeout_secs * and hung_task_check_interval_secs @@ -480,7 +500,7 @@ static const struct ctl_table hung_task_sysctls[] = { .data = &sysctl_hung_task_warnings, .maxlen = sizeof(int), .mode = 0644, - .proc_handler = proc_dointvec_minmax, + .proc_handler = proc_dohung_task_warnings, .extra1 = SYSCTL_NEG_ONE, }, { -- 2.55.0