From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39324C52D7B for ; Tue, 13 Aug 2024 11:04:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 471026B00A9; Tue, 13 Aug 2024 07:04:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3FA476B00AA; Tue, 13 Aug 2024 07:04:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 273506B00AC; Tue, 13 Aug 2024 07:04:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 045AE6B00A9 for ; Tue, 13 Aug 2024 07:04:37 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 800031A079D for ; Tue, 13 Aug 2024 11:04:37 +0000 (UTC) X-FDA: 82446938994.12.EC9CAA9 Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) by imf20.hostedemail.com (Postfix) with ESMTP id 820DB1C0024 for ; Tue, 13 Aug 2024 11:04:35 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=OHfqFg6Y; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.221.41 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723547063; a=rsa-sha256; cv=none; b=tRVT5oabPSZwpu+7ENVX39DOViCPTKStzws3vk4x/nV6Z4/+0KrAoP0iYsWJRP0HvIH6BX 3sJeW9Bs+vCssC7r0OPhZmAZ3MEjR9vMI+ueeTX++1CgJXHIVnt3C98IrSgzQ06bwc/0Ag TUXqJRaqNhr5Y5bp5CMLlSTs3mUplns= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=OHfqFg6Y; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.221.41 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723547063; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NRZJg0PF5C+Q/wLolSP2P0QFHIseDVqe88AMmYIa5J4=; b=ICB17sMvqj+V2IGb4J20jjE0GqGObGWCsaTPzvi7mrHYidUgDMmMUNa1ar9ffxZcTgsTER R4Os4uz9z1ayC2MpwfS0BJmMSzBhdbunXyHxh+xuxCOaCmDUQYCIBskGZiR8PG2ZIME8Ld I4xnRxBUVyKqJisTJXqXFWlNG4ZJHEw= Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-369f68f63b1so3091129f8f.2 for ; Tue, 13 Aug 2024 04:04:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723547074; x=1724151874; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=NRZJg0PF5C+Q/wLolSP2P0QFHIseDVqe88AMmYIa5J4=; b=OHfqFg6YMVfGy3Vrn6R6Tn7WauTVoIzXsjF3mgoega4uYXEz9+UvBZbawkdr5CPAUj lACKuAJg+qDKcIyseJdWBGBuxb13mjCA3XsxUHzv/U/NJrDy//TdhVM08C+A5Fkvl8zx wK67Mv4UtFvv1rf5FlRqsHfxf+B49JL6duezbvf6WK6XRJtYCP/5XtOGWs0RV7jFni5p L2jWRIubU6sqEOiHJ0LY4npsVrqrTMnPZkwYvJpO3Wxtg2YBC8RVwYnVixwOwNyb94pM 6Mb1fp1O5X2YY86FgeeIuoRi9S1a+Z+IHpKMogmGB4scPCMRuqOhSpiAB/nwXdd68SMK kj2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723547074; x=1724151874; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NRZJg0PF5C+Q/wLolSP2P0QFHIseDVqe88AMmYIa5J4=; b=jAWkOoa2icNLhU+U2lslJt3zWL2OoAGAiJKUFAe9pG46Uth9O9WAn/hE9s2sVPWxs4 1ZhBT7wXYEFJkga34cNTqbie8TVOKiVWvHEPevE+YV7ZF8tDm11dkdamp6nfSH5OYuW7 QqbsqSdta/udBhD8ZPMN94uaAuggVAYjMPibI6tXydzkbkbLuZ//rotwMgLn5A+lV2KC 0YKHHpU9OlMKTqG3/8HOjPfhYuerRKVG2bRDIUSyBkvm2QSsZPyaJH7sqvylBg9FHZhz yMvPhdRvFA3ifSIY4DNInBb/W+fUfuMObFiBRz0ykhYj8Xui3k6FWS1kmeVx/PhHAehC Xkug== X-Gm-Message-State: AOJu0YyortWijRO7WnzPAkxe9852ZNqZMV8jdIW+7cFD7nXhmRVPnZuE vM1xjidpl74O2rtRSBwGj/OFWs8VruJLkGyO6MwIXu3mqDEXofbp X-Google-Smtp-Source: AGHT+IEz839ACn+M9vUI6m6H0zHceIH7b/tAJn+ns5L4kLoKAUfv3y1iOdbbyqTyERxFRLCw45wb0A== X-Received: by 2002:a05:6000:1284:b0:368:4d33:9aac with SMTP id ffacd0b85a97d-3716ccfa3f5mr2037742f8f.31.1723547073508; Tue, 13 Aug 2024 04:04:33 -0700 (PDT) Received: from ?IPV6:2a02:6b6f:e750:7600:c5:51ce:2b5:970b? ([2a02:6b6f:e750:7600:c5:51ce:2b5:970b]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-36e4c938665sm9813034f8f.40.2024.08.13.04.04.32 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 13 Aug 2024 04:04:33 -0700 (PDT) Message-ID: <0702c85c-abca-4a33-b91b-dadf68670364@gmail.com> Date: Tue, 13 Aug 2024 12:04:32 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Hard and soft lockups with FIO and LTP runs on a large system To: Yu Zhao , Bharata B Rao Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, nikunj@amd.com, "Upadhyay, Neeraj" , Andrew Morton , David Hildenbrand , willy@infradead.org, vbabka@suse.cz, kinseyho@google.com, Mel Gorman , leitao@debian.org References: <1998d479-eb1a-4bc8-a11e-59f8dd71aadb@amd.com> <7a06a14e-44d5-450a-bd56-1c348c2951b6@amd.com> Content-Language: en-US From: Usama Arif In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Queue-Id: 820DB1C0024 X-Rspamd-Server: rspam01 X-Stat-Signature: bgucf1fza9xk5us6ygrxjcwfks55ezoo X-HE-Tag: 1723547075-748608 X-HE-Meta: U2FsdGVkX1/JRTX56q4VLAsYzZQ650X+dqLkU7yTlGnuqIwSCcpWqGbe9t2ikKF8GpJEykYkDUci7CX+FgidzHOPU7LtImR1sqeSrjpyIcsU50yyGWlBICCd8ICr7v51dnTK2s2AlAaYv8BpJ7NtjzrPc+P+LXTYo7JtIE+rGB3Jj2YfISRoVTlyqdMmAuASoWPvvy6wYur6AN+muHeqt1r8L//MG+T9w2UA0Uml2ux+OQ/FTHRRmc0uXYcf/J0dCSh9EjMksovPcgx15d3ScNFqIBKyoJPRFDwpRR0/Pu8Ucp1Wniqh+IHWuN9KgeMwICho1J3dvt6P3StQa227JV/DCEkYD5NH2WFk+7mieTElqgdXkLK1K/U53VdaliDyPv8gKGlXAZb4OzPnjicQLBFro1s577jt3PDlBmLzV47B/HKeVAiv7Z6en7KlWUV76b5390g2YgXSY0iiPJsyKK0bwffYhbaHoiOp18Li2sTLAQgJ3xhD/9Dk/iFeb3821lSin4r9f2Tbd5yDK1upkkejOMunVIc0D+D5pQrmXVx9WPEXahBBI1Vzdx+n51G9qdntLA2JY7GsEqSrQNcXL6zfS2aOPJfd+YM9pBt+XuSJDbO3ksW7rMkvT5cweNjBkYiBhYAFLPuIvPvXIkLo56WUVmuxTUy3oRXxoqoFBoiDMQZQ9E0YhTdz8AU7u4PpC3659rnvxPoUpVWlQ9nQ0EuS4ZCkBOx8Xl5ntJGJHsfAkqaQ7t6CLCDMlqd8+kbFo5QJVMx/mkwHWmTnK8VBc6ujraT3p/1s6b1Eot9RROOBbMOTvBtdmNmx/yOPrLWleBjxfMWlQP+rcTZYBNJfflFTNrRcUwT4Y0N7YTXplZSUpPTcwaTVD1qj50yKAXjOR1lbXk6eGxtncC08e8mO7O4bVmAHbxR6RHWNPaYbiVmf0pN76pGlVCK/AjWxxrfoIER1bB/G4hU+/Jkqx7V mEDEttxZ fjL+hmDYFuZSS9zh6g2RxxOeMCFJofV9WdKX1tNQ1h6DdVZeSp9cQ8/ASA8hdph5h684PSYohFiz7AXQGziNOIK2fEdSh7CpLF6atqT3SgWrEOrgYBpLFdrDs7z9QKXIVVYNtlvMwrIdAHh2k1jTclJLfozWt8V/PidaURnwW+ucpzdvUBA9m2ei5f9gTDKUxKDJK7GZRudCjKQCxrGAqzfpP7+jciiWjXJppwZw/G6UlAjVoa7zQCklMNqE9USkSIU+PlR8XOF1pz/SNfOG5sP6WCuZRL8gfLlgbCmocE4QngSzIgYSSQvJ2HybyVWo+5uFvyJ0F5Q2GK086PmY+BdkMMKRXZ8pYb9BGAbKq0+y4W6HVgBFKRiPbW4gqILY1YTfPLu5vTjpwHa7n5m8Tb7v30uyPy2pKEgRSfUx/lE4C5l71f9aPEitGQSCPAzMncgSBt+GuH/G9I94jj1yp5BeOKuczM5EGgbzfpKD6QKbFHRk0bgmU/OAMFfMcXI2sMKJwnOmoDl/Cotk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 09/07/2024 06:58, Yu Zhao wrote: > On Mon, Jul 8, 2024 at 10:31 PM Bharata B Rao wrote: >> >> On 08-Jul-24 9:47 PM, Yu Zhao wrote: >>> On Mon, Jul 8, 2024 at 8:34 AM Bharata B Rao wrote: >>>> >>>> Hi Yu Zhao, >>>> >>>> Thanks for your patches. See below... >>>> >>>> On 07-Jul-24 4:12 AM, Yu Zhao wrote: >>>>> Hi Bharata, >>>>> >>>>> On Wed, Jul 3, 2024 at 9:11 AM Bharata B Rao wrote: >>>>>> >>>> >>>>>> >>>>>> Some experiments tried >>>>>> ====================== >>>>>> 1) When MGLRU was enabled many soft lockups were observed, no hard >>>>>> lockups were seen for 48 hours run. Below is once such soft lockup. >>>>> >>>>> This is not really an MGLRU issue -- can you please try one of the >>>>> attached patches? It (truncate.patch) should help with or without >>>>> MGLRU. >>>> >>>> With truncate.patch and default LRU scheme, a few hard lockups are seen. >>> >>> Thanks. >>> >>> In your original report, you said: >>> >>> Most of the times the two contended locks are lruvec and >>> inode->i_lock spinlocks. >>> ... >>> Often times, the perf output at the time of the problem shows >>> heavy contention on lruvec spin lock. Similar contention is >>> also observed with inode i_lock (in clear_shadow_entry path) >>> >>> Based on this new report, does it mean the i_lock is not as contended, >>> for the same path (truncation) you tested? If so, I'll post >>> truncate.patch and add reported-by and tested-by you, unless you have >>> objections. >> >> truncate.patch has been tested on two systems with default LRU scheme >> and the lockup due to inode->i_lock hasn't been seen yet after 24 hours run. > > Thanks. > >>> >>> The two paths below were contended on the LRU lock, but they already >>> batch their operations. So I don't know what else we can do surgically >>> to improve them. >> >> What has been seen with this workload is that the lruvec spinlock is >> held for a long time from shrink_[active/inactive]_list path. In this >> path, there is a case in isolate_lru_folios() where scanning of LRU >> lists can become unbounded. To isolate a page from ZONE_DMA, sometimes >> scanning/skipping of more than 150 million folios were seen. There is >> already a comment in there which explains why nr_skipped shouldn't be >> counted, but is there any possibility of re-looking at this condition? > > For this specific case, probably this can help: > > @@ -1659,8 +1659,15 @@ static unsigned long > isolate_lru_folios(unsigned long nr_to_scan, > if (folio_zonenum(folio) > sc->reclaim_idx || > skip_cma(folio, sc)) { > nr_skipped[folio_zonenum(folio)] += nr_pages; > - move_to = &folios_skipped; > - goto move; > + list_move(&folio->lru, &folios_skipped); > + if (spin_is_contended(&lruvec->lru_lock)) { > + if (!list_empty(dst)) > + break; > + spin_unlock_irq(&lruvec->lru_lock); > + cond_resched(); > + spin_lock_irq(&lruvec->lru_lock); > + } > + continue; > } > Hi Yu, We are seeing lockups and high memory pressure in Meta production due to this lock contention as well. My colleague highlighted it in https://lore.kernel.org/all/ZrssOrcJIDy8hacI@gmail.com/ and was pointed to this fix. We removed skip_cma check as a temporary measure, but this is a proper fix. I might have missed it but didn't see this as a patch on the mailing list. Just wanted to check if you were planning to send it as a patch? Happy to send it on your behalf as well. Thanks