From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33207C52D7C for ; Tue, 13 Aug 2024 09:49:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 67DD16B007B; Tue, 13 Aug 2024 05:49:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 62D906B00A9; Tue, 13 Aug 2024 05:49:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F5036B00AA; Tue, 13 Aug 2024 05:49:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2F8656B00A9 for ; Tue, 13 Aug 2024 05:49:54 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8FBE6C0721 for ; Tue, 13 Aug 2024 09:49:53 +0000 (UTC) X-FDA: 82446750666.17.AC18CDB Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) by imf14.hostedemail.com (Postfix) with ESMTP id 8C79D100008 for ; Tue, 13 Aug 2024 09:49:51 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf14.hostedemail.com: domain of breno.debian@gmail.com designates 209.85.218.41 as permitted sender) smtp.mailfrom=breno.debian@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723542539; a=rsa-sha256; cv=none; b=FIJO0N/tMNVD36wz9SLnfV4gXLrbpC0dHr+FDVZbYv7jbDQdNohI7R0BkCIpkAtyzCFfdZ q2fEWnqHftmj+KoNGtO7bTah/OZG0BKaCusmQ5U+X7HTKRn0dz9CT0jbH9el/5CRN5rc15 Ez6PAseKkiak4M0jOHH06QYJgj6m1KU= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf14.hostedemail.com: domain of breno.debian@gmail.com designates 209.85.218.41 as permitted sender) smtp.mailfrom=breno.debian@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723542539; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fbqSnnhU9hTXfKatLvVtC315COj3Ikq+Kw2F310p32M=; b=S8MVue3odayPA86xjUJvfUW3ZYIHQ38Ww2Z7uoRB6P5hV3RK9ly3ixv7WE+/gljCFrEe5o H37VrU1qyh1bnFcULBKzp64Zzwjlcez15CCEnC9EHlDdIM5NWrdNbQ8SkfRgLHVmnaaxks LYPkCa8AxG23yfzLShKQvUVWZgKX0Ts= Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-a7a83a968ddso514008466b.0 for ; Tue, 13 Aug 2024 02:49:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723542590; x=1724147390; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=fbqSnnhU9hTXfKatLvVtC315COj3Ikq+Kw2F310p32M=; b=ZbnODW3NTlksnE/19s0HZvqqYp4PB+Me8giSFIgnOdHwdRzCsJi0bljeEIroaHsHcA G0yNAM0K1rOlXBu4DY8/tywiysCkypbmtY2m53tbBvuXOWSlz9qran0tj/Wq5TANeinz 7THBbGLAPX/LYBGpDYChYWtVbx9l+SCOYcvCmy5+Gxb9uKAmNgdNBuC7JAP0hWwg4q9J QqQIyhtIqAOYfTi3odIG715P/4hhb70bVu8WZWq8r2aged68QfH9Z2Wy1TTYPoCbCjn/ k+6QXq86GIrlihyp6GAau6u8o2Pdu6hga8SUVSExQ1QpI4rHb4ALsx29+hQQWdVEQzf5 mO1w== X-Forwarded-Encrypted: i=1; AJvYcCUm1m/TCdA3kli12J5Ro7h0OuLDKAVrTDHCb9lALlu9Y+JbzNOpWLVyfnYx5fmguO9iE8OukKdVrjFv4HCifQ8o5bw= X-Gm-Message-State: AOJu0Yzv7yTHMKIEtRvVk71bX10+7YMOvxsGby/mUsu030Hf+p9F0WaD rXOe7VccbGclZQavwJF+SGMSeGBG6doSOHGK3LKNM47PcD95xULF X-Google-Smtp-Source: AGHT+IF2hWJHoMLs1hIwGm6kNwLPGYBD/pRH5khWiBaBIGwyKPbl1wBOloRX+ZvW7jVB556SJCfi0A== X-Received: by 2002:a17:907:608a:b0:a7d:391f:17af with SMTP id a640c23a62f3a-a80ed2c4690mr183106966b.51.1723542589170; Tue, 13 Aug 2024 02:49:49 -0700 (PDT) Received: from gmail.com (fwdproxy-lla-115.fbsv.net. [2a03:2880:30ff:73::face:b00c]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a80f414bbf3sm55291766b.145.2024.08.13.02.49.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Aug 2024 02:49:48 -0700 (PDT) Date: Tue, 13 Aug 2024 02:49:46 -0700 From: Breno Leitao To: "zhaoyang.huang" Cc: Andrew Morton , Matthew Wilcox , Suren Baghdasaryan , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Zhaoyang Huang , ke.wang@unisoc.com, usamaarif642@gmail.com, riel@surriel.com, hannes@cmpxchg.org, nphamcs@gmail.com Subject: Re: [PATCHv5] mm: skip CMA pages when they are not available Message-ID: References: <1685501461-19290-1-git-send-email-zhaoyang.huang@unisoc.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1685501461-19290-1-git-send-email-zhaoyang.huang@unisoc.com> X-Rspamd-Queue-Id: 8C79D100008 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 3bdem6d67wmo5rs6r6g8y8twidkxxx8m X-HE-Tag: 1723542591-195882 X-HE-Meta: U2FsdGVkX1+hyhLbR89DwC/jTVou4le8JTvMMUt5pO4pWHjEkFx33FagapMFolCQtm5Ocl+tkFQftHcpP8sr2OX57w3NVyxmvLh9HxZsqsiGv14+oDQ8Iw1MZkAvSL9in5erLZZ40yXfMkjLZU10XJcw6inf4r/1EhwITmqpxcTx/lnHgQ1HvFpYQTBKFYCxIGm3HVr6uozk2bfnGm0Z5T61KHmMngAgUeYsX1420EztvUZsMkJvZ/xy7oD6WzAgzm0VI1LG2Tmw9giFdFwp2RxWvRq9tlolg12Gal/K6+VsykhHqnFo3TMijTpEe+nbCMsqOeZBJfipPInUwoK2geyd6CwiNvJQkKQWtmTL+PeYriapCfKhfAMEmQcYx8Ka/RQPCJs9b1KoOCRs9lp1Lnru3kyeZO7TFwvrmwh0zv6HF1+n70hRG1pla73u8s0XIox+8s7yslSMxdUTiBFOwvUq/U8wVAqZhTMQgjlz7mlxTbWtHJJXI4wfmQ3BPHb46TVRF4uywfuBTV24Y09GCWz89fT3GbwoiD1UXgEyltuHxnPe0NyY6hm074CoIIgN7FUBBFkmOrWskk+e6N7uKPZ1uWGCc9z58X9fFl/ToCltFNrcMnsuUjSVgmY4T3a3u5P33enPT+9rwR4nkPcW55f/a9HPKRYsO3cajpqRnD2XsTWYoifBjND+0NrOJgelO5lFLUVOKXOe5kiQnotc1WTbXYDpIxrHAW8cdUmfQVqSTmgHpPknmV+xQIQtsjt+T4U9cTHsR2SLygGgYlJg5mNUuMRKKfdPL72Q376DGsqKYpn4JUm9LlUhX978NMSI7y96imDgxWyclp9g0fNN39BMob7BG3jjjcK5Rr5zzjKR6qZVi63Z/vSQpqTyWDuuXEb2mEKtA/8kka8smOZa6o8kiM3+EqKGO1TJ+fb0/5HcUg/3onv2+kq7gJ01YJN+pyxAv6bEgNYcz4G/R5j 2nwPBRUh Bo+3+Za/1up80GJ9vuvJdKMdEHoB5voLxJJ9foUepsRiShwNai3SAiTXBvqD9vHW9QaiIEXFknkL2yIZUnmYBZSgxKxpNJQsYoaK9bKHGBx4Oc00FexbBfz1CGbx/p/niK4L7vohoYbLEZzTOFw+1b+r1tdzowKo1TvHX/wkE63/JNEqDIJrpMbMIYEuQItOAdUkLmzLvmjLH4yCuwCdQh11HHSwYiHoONk4KMZsaNqfsc8BhsE5cppF3zDITCQKqGbwXUKHlu1ybnRXkqZcI4Lcx/u3CuHTpe1rNXoqnRhDTllWwJySKdnKI+osRV8WkZGhhfVDbSWQzQ5qERBon2uKqooZYBK8RDmWYtsnfEW30e4+0OPp8FzGOvmCpWIHC7/44 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, May 31, 2023 at 10:51:01AM +0800, zhaoyang.huang wrote: > From: Zhaoyang Huang > > This patch fixes unproductive reclaiming of CMA pages by skipping them when they > are not available for current context. It is arise from bellowing OOM issue, which > caused by large proportion of MIGRATE_CMA pages among free pages. Hello, I've been looking into a problem with high memory pressure causing OOMs in some of our workloads, and it seems that this change may have introduced lock contention when there is high memory pressure. I've collected some metrics for my specific workload that suggest this change has increased the lruvec->lru_lock waittime-max by 500x and the waittime-avg by 20x. Experiment ========== The experiment involved 100 hosts, each with 64GB of memory and a single Xeon 8321HC CPU. The experiment ran for over 80 hours. Half of the hosts (50) were configured with the patch reverted and lock stat enabled, while the other half was run against the upstream version. All machines had hugetlb_cma=6G set as a command-line argument. In this context, "upstream" refers to kernel release 6.9 with some minor changes that should not impact the results. Workload ======== The workload is a Java based application that fully utilized the memory, in fact, the JVM runs with `-Xms50735m -Xmx50735m` arguments. Results: ======= A few values from lockstat: waittime-max waittime-total waittime-avg holdtime-max 6.9: 242889 15618873933 715 17485 6.9-with-revert: 487 688563299 34 464 The full data could be seen at: https://docs.google.com/spreadsheets/d/1Dl-8ImlE4OZrfKjbyWAIWWuQtgD3fwEEl9INaZQZ4e8/edit?usp=sharing Possible causes: ================ I've been discussing this with colleagues and we're speculating that the high contention might be linked to the fact that CMA regions are now being skipped. This could potentially extend the duration of the isolate_lru_folios() 'while' loop, resulting in increased pressure on the lock. However, I want to emphasize that I'm not an expert in this area and I am simply sharing the data I collected.