From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA7EE38F23C for ; Thu, 5 Feb 2026 10:54:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770288893; cv=none; b=TJrr6T5YvXAGTqmA5aXgQJSncwvVYtobolW6dGyJGThL6H20WcWqUGCEuvQJB916HbKsIwrjObsk6zn0pUhUSWpUH25AF8oDkwp/RK11ktSrAtxCv8uZgzKhx6fUYtCqkkfHVu9VEQW8VeYDymHW6p5YU4fECl2+rVkGm+dr/cY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770288893; c=relaxed/simple; bh=UzIJ5uLmy6/QZkt4fUvhxcZm9S704OaZSpcQq971ZZs=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=ejoHuSeVLPChGsHiSsQ88JAiAx8uyyCbEK2/TShKMzKIxHYzOisnUsqSVEFkkctFqRzNZWcMLpBCdLkgLQkF0e9yTanMt3Pd817lewWBd6aKsiEhTJj2b/v+F6zhFIj4GdAQDoP10vM9blTJtJh7TkDiXhQp79uVv1Ci4CGWAso= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=fromorbit.com; spf=pass smtp.mailfrom=fromorbit.com; dkim=pass (2048-bit key) header.d=fromorbit-com.20230601.gappssmtp.com header.i=@fromorbit-com.20230601.gappssmtp.com header.b=DX+QH964; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=fromorbit.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fromorbit.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=fromorbit-com.20230601.gappssmtp.com header.i=@fromorbit-com.20230601.gappssmtp.com header.b="DX+QH964" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-81f5381d168so938255b3a.2 for ; Thu, 05 Feb 2026 02:54:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fromorbit-com.20230601.gappssmtp.com; s=20230601; t=1770288893; x=1770893693; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=R73q+WeQhTVseqbSpyKGYx71Ii0OR2nQn6ZRCzszols=; b=DX+QH9648nrjLauuvP7r/zlUDamx4RFL6PYlJGAbun1ljPrgPksQRaDaPs6d526OYU eAfZlgJgpy6tEFI6iBHa51gnKFoes0CEt1sp9FgxQYfSybdAY6bGBN2RsPl6GZFvX4vT kejsk7mS5/Em+CrxEOzlzYrGWv099qGLuNqJLprlmIlGx3TeoBpdp4Qs4U3I0iGodSc+ XKXpYmM/4obR5+ul5W3qivkgv7PjweXnECj6tUz1iPms+/hlY35PXGCJfkAYnF0qaZue icb8k/qgzT51Tt5sAPgsNn4iITKqQzx5WCF6iAkqMDTLnUJUkgCEAyQrB3TsVj3ztb0M x9DA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1770288893; x=1770893693; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=R73q+WeQhTVseqbSpyKGYx71Ii0OR2nQn6ZRCzszols=; b=Mw9KYINjQNDL7dDDsk4FHxChNs5SsucqNpQuSK31yyttXHVrc1z41eafnhNXJ7cxud 3M+pRMBHuzpyJaFfdheiTqTcLqFfInzWMkC+Zn5az6RNRmo7Jd+yZTr9NNnPlxYQLY02 /KfXaGyIvAICAVNLQ7Z1joFRtuLBwGr8LWhE5zG0tFVHyV0NM8ZvweUcy0QVAS3Le/i0 vaFcP7KvsAf0KWSp7ro5wJ7vRjE+Q7VhAhOOn6UJjgV0OKkjU3fw4BPdiqY3O8atlFRo S4qxpH0sbLGGhx5gXcjaEZD2gm2kmjcMpv8GGcMeRLNpsCs3oXJaq5P8eGNb2ylsT3as slJw== X-Forwarded-Encrypted: i=1; AJvYcCVEXM03olajsQbN8Mp3iQ9WOVhDqHgYx+Cpzhx1WAadl4on+JSVrvGoN6CSWcppE8ckayg61y5RSEtY2r0=@vger.kernel.org X-Gm-Message-State: AOJu0YzeAtcF2cA7GdHlr0RbFXZAznw22rk4TJl0iMki2aWS+8q+J/i4 xwJxj7OSP61EcazD6ExBafqKMT3O55a3P5X4wn4AIJgF/Baw5Ly4t3svVpUwG/D2aT8= X-Gm-Gg: AZuq6aIEAkeXDLGygHnBmOj3C6fZ5gZcZSddgtx9ykYG+7RkAB2VyLzxzZs0Hu90SjT rmI5ZvIvqWTzw2HNLsBWwdUaEmVxDyVNujG0BovVKBEtkl7HJO934tVdqJ3mQUsC5XL1uSsWzL9 2YvU4fJ/T9n9F30D6Tn5fwb9Fuj/OGEbFCFgAeZ6gJcBI1QQ2LRAyY9QVo5ALg4TBEQ3NgKc8Mq tKVoA6i+m0IV8eZDrZCe/7nL7N3j125YfvvLpdeF9Ql6UIoHyiZNoLx8xnLf8wbuCxzc6aE9BEP Rm1pFMXdczTuYxbVB1q1N7cySgnIInGGXmYIzz1b6hxLJtUtHaHDrfQl5DvoP7kCxjQsTYcv9P9 f/59My4stv26JJkoieJQrWuqiGGgiMf63UDzD/t/6W/rNv/OcwaEKQ8fp+daiT6OOra5ZrkRZS+ BEtwdZs0xJZtz2UDpX4kXN9GcBpmgIJ06UrvFpTc/ddJ1zgZ/myFoWI8kWQ7ZDfXQ= X-Received: by 2002:a05:6a21:33a3:b0:38b:dd94:936d with SMTP id adf61e73a8af0-393720d7387mr6307529637.21.1770288893040; Thu, 05 Feb 2026 02:54:53 -0800 (PST) Received: from dread.disaster.area (pa49-180-164-75.pa.nsw.optusnet.com.au. [49.180.164.75]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-c6c8581fe1esm4602582a12.35.2026.02.05.02.54.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Feb 2026 02:54:52 -0800 (PST) Received: from dave by dread.disaster.area with local (Exim 4.99.1) (envelope-from ) id 1vnx0b-0000000DD9B-32ae; Thu, 05 Feb 2026 21:54:49 +1100 Date: Thu, 5 Feb 2026 21:54:49 +1100 From: Dave Chinner To: alexjlzheng@gmail.com Cc: cem@kernel.org, linux-xfs@vger.kernel.org, linux-kernel@vger.kernel.org, Jinliang Zheng Subject: Re: [PATCH 1/2] xfs: take a breath in xlog_ioend_work() Message-ID: References: <20260205082621.2259895-1-alexjlzheng@tencent.com> <20260205082621.2259895-2-alexjlzheng@tencent.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260205082621.2259895-2-alexjlzheng@tencent.com> On Thu, Feb 05, 2026 at 04:26:20PM +0800, alexjlzheng@gmail.com wrote: > From: Jinliang Zheng > > The xlog_ioend_work() function contains several nested loops with > fairly complex operations, which may leads to: > > PID: 2604722 TASK: ffff88c08306b1c0 CPU: 263 COMMAND: "kworker/263:0H" > #0 [ffffc9001cbf8d58] machine_kexec at ffffffff9d086081 > #1 [ffffc9001cbf8db8] __crash_kexec at ffffffff9d20817a > #2 [ffffc9001cbf8e78] panic at ffffffff9d107d8f > #3 [ffffc9001cbf8ef8] watchdog_timer_fn at ffffffff9d243511 > #4 [ffffc9001cbf8f28] __hrtimer_run_queues at ffffffff9d1e62ff > #5 [ffffc9001cbf8f80] hrtimer_interrupt at ffffffff9d1e73d4 > #6 [ffffc9001cbf8fd8] __sysvec_apic_timer_interrupt at ffffffff9d07bb29 > #7 [ffffc9001cbf8ff0] sysvec_apic_timer_interrupt at ffffffff9dd689f9 > --- --- > #8 [ffffc900460a7c28] asm_sysvec_apic_timer_interrupt at ffffffff9de00e86 > [exception RIP: slab_free_freelist_hook.constprop.0+107] > RIP: ffffffff9d3ef74b RSP: ffffc900460a7cd0 RFLAGS: 00000286 > RAX: ffff89ea4de06b00 RBX: ffff89ea4de06a00 RCX: ffff89ea4de06a00 > RDX: 0000000000000100 RSI: ffffc900460a7d28 RDI: ffff888100044c80 > RBP: ffff888100044c80 R8: 0000000000000000 R9: ffffffffc21e8500 > R10: ffff88c867e93200 R11: 0000000000000001 R12: ffff89ea4de06a00 > R13: ffffc900460a7d28 R14: ffff89ea4de06a00 R15: ffffc900460a7d30 > ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 > #9 [ffffc900460a7d18] __kmem_cache_free at ffffffff9d3f65a0 > #10 [ffffc900460a7d70] xlog_cil_committed at ffffffffc21e85af [xfs] > #11 [ffffc900460a7da0] xlog_cil_process_committed at ffffffffc21e9747 [xfs] > #12 [ffffc900460a7dd0] xlog_state_do_iclog_callbacks at ffffffffc21e41eb [xfs] > #13 [ffffc900460a7e28] xlog_state_do_callback at ffffffffc21e436f [xfs] > #14 [ffffc900460a7e50] xlog_ioend_work at ffffffffc21e6e1c [xfs] > #15 [ffffc900460a7e70] process_one_work at ffffffff9d12de69 > #16 [ffffc900460a7ea8] worker_thread at ffffffff9d12e79b > #17 [ffffc900460a7ef8] kthread at ffffffff9d1378fc > #18 [ffffc900460a7f30] ret_from_fork at ffffffff9d042dd0 > #19 [ffffc900460a7f50] ret_from_fork_asm at ffffffff9d007e2b > > This patch adds cond_resched() to avoid softlockups similar to the one > described above. You've elided the soft lockup messages that tell us how long this task was holding the CPU. What is the soft lockup timer set to? What workload causes this to happen? How do we reproduce it? FWIW, yes, there might be several tens of thousands of objects to move to the AIL in this journal IO completion path, but if this takes more than a couple of hundred milliseconds of processing time then there is something else going wrong.... -Dave. -- Dave Chinner david@fromorbit.com