From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81B95C3271E for ; Mon, 8 Jul 2024 16:18:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 15D776B00A5; Mon, 8 Jul 2024 12:18:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 10DB16B00A6; Mon, 8 Jul 2024 12:18:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F16FF6B00A8; Mon, 8 Jul 2024 12:18:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D377A6B00A5 for ; Mon, 8 Jul 2024 12:18:13 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 70EC380CC6 for ; Mon, 8 Jul 2024 16:18:13 +0000 (UTC) X-FDA: 82317092466.13.C85B59F Received: from mail-qt1-f178.google.com (mail-qt1-f178.google.com [209.85.160.178]) by imf10.hostedemail.com (Postfix) with ESMTP id A8C60C000E for ; Mon, 8 Jul 2024 16:18:11 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=y2a1VqZF; spf=pass (imf10.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.178 as permitted sender) smtp.mailfrom=yuzhao@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720455461; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1PBAQem0cF7wTCAm7Va1RSuppap0iOvlnk+12A4PzYE=; b=pNJ4IvFdj7+xoANuql/nzRgIj/tNNjDOpEXJgzpN/4FeujktrV8FG5ZqdAiXdHyJNxRZ92 HDeq+Q8YlDbiwI+b4WAfUWdxwScT5n2yDQhj7drbjWLuXWwB9jiVKxp5dkixzNF6L2kbSD wfSXSZj0lBp4p17UWPaCzLj/aYGe76U= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720455461; a=rsa-sha256; cv=none; b=tGKXQRwmgEz+NSU0t8yhGbIuXwAYfbsAdsQO6WfuVG5R2JzhVjne4Dhj+BPpGJQpMff2bZ eQIJCEBjHHIl5h4X4Mr1kW1QDjjVuqKvq4QFEOlzJJK7unwo5wXOfi4QNO1IUeWVLppO8e eIhfBq0PxDUsz7UrmltHndCdu8708XE= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=y2a1VqZF; spf=pass (imf10.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.178 as permitted sender) smtp.mailfrom=yuzhao@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-qt1-f178.google.com with SMTP id d75a77b69052e-447dabd9562so695891cf.0 for ; Mon, 08 Jul 2024 09:18:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720455491; x=1721060291; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=1PBAQem0cF7wTCAm7Va1RSuppap0iOvlnk+12A4PzYE=; b=y2a1VqZFBBGtLr7YMP1nCJN4HjNj5pEQs3IPSzI8sdPV0UpNqw+zuFZYLfyxYfYpZo vbWwiyaQIYnz7boag2NywyFW8SVetxTcRXgvV6XmIHnXq5UG1cJvzEJLxSt2ru+TPNiO mlNEUr2dfLNArWzu/ky5+HR/F3Iq3qlMU0RGPBERL3TwQiw+Ju792aMylMsjfK2cot9z AngiqPkmwokliqCJKttHZskC5yEWDFcIzSF36LB+cyhsW+BJ6EIcv8G7vJP5fEz7xzTH mnUYxtBgbtMQ1m9GF9UO9JY5II611An1j7eNOOmXF0pm7MXrxhHxNCoolMmdrS2nstW5 jAXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720455491; x=1721060291; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1PBAQem0cF7wTCAm7Va1RSuppap0iOvlnk+12A4PzYE=; b=oB7+Nx4iHlAzqXfRWo/zd6iCqZTA9XlxlD6NiZ7CFQTsvh8jhgE+XM8EyA8GDq8tK6 Cx5AS2XQtprxkAhmB6GXRS1Zg21ZUtDm5FfERolmIlVkKglaJMM2qDWL+bZG7XqOJ1fG 4YZf9bqQtJgEfxZ1UuUp0d2OgvZOyqN/2/YgpMbRlLfTZpi9yLru6yg05nRiYW61aVcG 6Hipjb8Nl63mnEiW4dbvDfVjogCiCwyhzLcx6Ei7fhMUn4ebYdg1k+0qfiZjXATTHArS I8WcsPSIhwl63ePuYDZWcJm9e0q8ICMXTyIhZT+1NxAehvw0GPSx+JJw+U/tbiQs0+Pk SQaw== X-Gm-Message-State: AOJu0YznokzM/szAOvd0pwWcRSoDW4Kkg/hnOitNVHq0DcpLZE81+FIN Y28bBHOsAc/3dRjvm8ze+6T29nkzlhKAnwpJzjMafs+iN2Ifm5izqUZcG6ns5OfBi0OBA5iM4CU L8Odw1GFtJg2vZqMkzfFZ3XNBR4G1KQTIAO7A X-Google-Smtp-Source: AGHT+IHi+IWF1ZgStNdLQo8ikEX6DJSqCsuQ043sS/L3Us0b98GZnJwgAviB24/eaMcsMz8+p039RkyuN07FEV22Yq0= X-Received: by 2002:ac8:7ef7:0:b0:447:f914:8719 with SMTP id d75a77b69052e-447f9148dc8mr668261cf.2.1720455490297; Mon, 08 Jul 2024 09:18:10 -0700 (PDT) MIME-Version: 1.0 References: <1998d479-eb1a-4bc8-a11e-59f8dd71aadb@amd.com> In-Reply-To: <1998d479-eb1a-4bc8-a11e-59f8dd71aadb@amd.com> From: Yu Zhao Date: Mon, 8 Jul 2024 10:17:31 -0600 Message-ID: Subject: Re: Hard and soft lockups with FIO and LTP runs on a large system To: Bharata B Rao Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, nikunj@amd.com, "Upadhyay, Neeraj" , Andrew Morton , David Hildenbrand , willy@infradead.org, vbabka@suse.cz, kinseyho@google.com, Mel Gorman Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: h78zi5rnm6f4jht6u994w8xepcgq7ubp X-Rspamd-Queue-Id: A8C60C000E X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1720455491-641937 X-HE-Meta: U2FsdGVkX1/DRS4thV0j2ztRMGUenRwVRf7N8mbEYvWW+izBm9F7hKywkRgraYux445GD/by54CzrNITFvKXfEWVc1LNmPMJXWHqiIiHET+/F52z52A9UySwNDVSmpTfgqyT4RZ+gKKAtQ6z+IFovuah+24YmGRDkaPjgNlh3/Vm8Wulliz8vblE1yzEuOAw4DRU7lwVdS2PSN0o/9jVv7UKBYEeEyvdMnTcwy7p/0Dsr37HjRY1vaO0W3OmEOc6IZL5rBl8PYlXsrLZR5/e+jf7PcjkvpJUkX2vrLb3aYMr9heAsVQ7n/9khejFWOj2SxNLMZrFF97LhpVPgxc72WVAY5IHpldVjZS93h70A8Tuu1IRxDd9q4+H8viwS8l2FoD+B5JwzdmBi5gGuzNjpG4eoPWnRRqvU6/I/gv594Ivud3WZb6ard7L2FRMN16yYPTIkr3OgYocVCU7yixtWa+tX3LZahl9pHbo3je4q8qHp6I3kQBKYhOc8Sr3I4BS45EJqFHhDKqVnFNV/mUbXHALc+aotlWaCcu2Hst97muG8//Oas/2rKOxpW+Zf+tcP/gVPZ7fZEiqRm7pQSl0wpWbw5ClxEGvnF/5KueD1qThY/YrOAb3MqzXXsjni3878OTbT92lo3ewWKweJIjCwrVutHSI0ldhiZomPcAsoKJfZG1eBGjrCjH5tJSZCpKM5LeVTByh96YMgSMyl++DTK1yije01a5Dm208OAotB+3KDCLEvAFzjCsP0haxxONj1anxYTK3mrqktRytoKJrMwEKfJoVGS9MMSAXCaXLvWbyG1MKPYZj4h2xI8NChZs4/0WFRqh0Fzi6q0C5X5ioRB/tPUO3+MUb/Mf3tGRt1j2kFzpTRq72fQX7JNuxNea7ESgxnJY0k6gkpaJV0uwRWuXIPsdsCHStYGPrfGDaBbY5hkx/gCkJwSLxVL6IqRSBYD6puYnhT7hIu57EEWg 9tl8jj/1 ipoGMzOGfl4bMGpV3jXp8MIiJQMSMxhMMXHxGkS0N1FXjiNV406504dx7OXm1l4ea0SerTmxxb0No4NFxpnvgoa1et5FrtM11Ra+dBNMMX3hHvSTI37/zqBAljpY7RhzgIqbpmK0OXo3uO/eBii9XPFbmF/u1luA+gRmQf+EXGoY5YI4elDxgt6yPuwoD6PbUKP7TO67WVlhidS8yR3KyBESEU7mRktp8+e6Mi+MSY5uf+Tk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000335, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Jul 8, 2024 at 8:34=E2=80=AFAM Bharata B Rao wrot= e: > > Hi Yu Zhao, > > Thanks for your patches. See below... > > On 07-Jul-24 4:12 AM, Yu Zhao wrote: > > Hi Bharata, > > > > On Wed, Jul 3, 2024 at 9:11=E2=80=AFAM Bharata B Rao = wrote: > >> > > >> > >> Some experiments tried > >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > >> 1) When MGLRU was enabled many soft lockups were observed, no hard > >> lockups were seen for 48 hours run. Below is once such soft lockup. > > > > This is not really an MGLRU issue -- can you please try one of the > > attached patches? It (truncate.patch) should help with or without > > MGLRU. > > With truncate.patch and default LRU scheme, a few hard lockups are seen. Thanks. In your original report, you said: Most of the times the two contended locks are lruvec and inode->i_lock spinlocks. ... Often times, the perf output at the time of the problem shows heavy contention on lruvec spin lock. Similar contention is also observed with inode i_lock (in clear_shadow_entry path) Based on this new report, does it mean the i_lock is not as contended, for the same path (truncation) you tested? If so, I'll post truncate.patch and add reported-by and tested-by you, unless you have objections. The two paths below were contended on the LRU lock, but they already batch their operations. So I don't know what else we can do surgically to improve them. > First one is this: > > watchdog: Watchdog detected hard LOCKUP on cpu 487 > CPU: 487 PID: 11525 Comm: fio Not tainted 6.10.0-rc3 #27 > RIP: 0010:native_queued_spin_lock_slowpath+0x81/0x300 > Call Trace: > > ? show_regs+0x69/0x80 > ? watchdog_hardlockup_check+0x1b4/0x3a0 > > ? native_queued_spin_lock_slowpath+0x81/0x300 > > > ? __pfx_folio_activate_fn+0x10/0x10 > _raw_spin_lock_irqsave+0x5b/0x70 > folio_lruvec_lock_irqsave+0x62/0x90 > folio_batch_move_lru+0x9d/0x160 > folio_activate+0x95/0xe0 > folio_mark_accessed+0x11f/0x160 > filemap_read+0x343/0x3d0 > > blkdev_read_iter+0x6f/0x140 > vfs_read+0x25b/0x340 > ksys_read+0x67/0xf0 > __x64_sys_read+0x19/0x20 > x64_sys_call+0x1771/0x20d0 > > This is the next one: > > watchdog: Watchdog detected hard LOCKUP on cpu 219 > CPU: 219 PID: 2584763 Comm: fs_racer_dir_cr Not tainted 6.10.0-rc3 #27 > RIP: 0010:native_queued_spin_lock_slowpath+0x2b4/0x300 > Call Trace: > > ? show_regs+0x69/0x80 > ? watchdog_hardlockup_check+0x1b4/0x3a0 > > ? native_queued_spin_lock_slowpath+0x2b4/0x300 > > > _raw_spin_lock_irqsave+0x5b/0x70 > folio_lruvec_lock_irqsave+0x62/0x90 > __page_cache_release+0x89/0x2f0 > folios_put_refs+0x92/0x230 > __folio_batch_release+0x74/0x90 > truncate_inode_pages_range+0x16f/0x520 > truncate_pagecache+0x49/0x70 > ext4_setattr+0x326/0xaa0 > notify_change+0x353/0x500 > do_truncate+0x83/0xe0 > path_openat+0xd9e/0x1090 > do_filp_open+0xaa/0x150 > do_sys_openat2+0x9b/0xd0 > __x64_sys_openat+0x55/0x90 > x64_sys_call+0xe55/0x20d0 > do_syscall_64+0x7e/0x130 > entry_SYSCALL_64_after_hwframe+0x76/0x7e > > When this happens, all-CPU backtrace shows a CPU being in > isolate_lru_folios(). > > > > >> kernel: watchdog: BUG: soft lockup - CPU#29 stuck for 11s! [fio:270164= 9] > >> kernel: CPU: 29 PID: 2701649 Comm: fio Tainted: G L > >> 6.10.0-rc3-mglru-irqstrc #24 > >> kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x2b4/0x300 > >> kernel: Call Trace: > >> kernel: > >> kernel: ? show_regs+0x69/0x80 > >> kernel: ? watchdog_timer_fn+0x223/0x2b0 > >> kernel: ? __pfx_watchdog_timer_fn+0x10/0x10 > >> > >> kernel: > >> kernel: > >> kernel: ? asm_sysvec_apic_timer_interrupt+0x1b/0x20 > >> kernel: ? native_queued_spin_lock_slowpath+0x2b4/0x300 > >> kernel: _raw_spin_lock+0x38/0x50 > >> kernel: clear_shadow_entry+0x3d/0x100 > >> kernel: ? __pfx_workingset_update_node+0x10/0x10 > >> kernel: mapping_try_invalidate+0x117/0x1d0 > >> kernel: invalidate_mapping_pages+0x10/0x20 > >> kernel: invalidate_bdev+0x3c/0x50 > >> kernel: blkdev_common_ioctl+0x5f7/0xa90 > >> kernel: blkdev_ioctl+0x109/0x270 > >> kernel: x64_sys_call+0x1215/0x20d0 > >> kernel: do_syscall_64+0x7e/0x130 > >> > >> This happens to be contending on inode i_lock spinlock. > >> > >> Below preemptirqsoff trace points to preemption being disabled for mor= e > >> than 10s and the lock in picture is lruvec spinlock. > > > > Also if you could try the other patch (mglru.patch) please. It should > > help reduce unnecessary rotations from deactivate_file_folio(), which > > in turn should reduce the contention on the LRU lock for MGLRU. > > Currently testing is in progress with mglru.patch and MGLRU enabled. > Will get back on the results. Thank you.