From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 052EAC3DA60 for ; Wed, 17 Jul 2024 16:34:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 92B366B0099; Wed, 17 Jul 2024 12:34:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DBB06B009A; Wed, 17 Jul 2024 12:34:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7A3846B009B; Wed, 17 Jul 2024 12:34:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 59B296B0099 for ; Wed, 17 Jul 2024 12:34:09 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 0AA3A40AED for ; Wed, 17 Jul 2024 16:34:09 +0000 (UTC) X-FDA: 82349791818.21.C0B68C0 Received: from mail-lj1-f177.google.com (mail-lj1-f177.google.com [209.85.208.177]) by imf18.hostedemail.com (Postfix) with ESMTP id 198CD1C0025 for ; Wed, 17 Jul 2024 16:34:06 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WdufjRKj; spf=pass (imf18.hostedemail.com: domain of kmanaouil.dev@gmail.com designates 209.85.208.177 as permitted sender) smtp.mailfrom=kmanaouil.dev@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721234016; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=32qbVbdcIIo082PD1NEo17pI9Fe6+jCMt4sWhJ+oEKQ=; b=GE350A5yAOo8oxlqbTLIlMhV4RoN93SWg6PZoavij2C2wisXGqb4LSdzJtFTUyP/nzQjbS 5W4M2X9+2IDzmbSbVkxCUqpvd7pdAceZlEVWBhMqS4sxZ9Q37/XHdf9sdPKUU8e7IxEvRJ hCXOg6sewSMxZsc/2umHTwuEVTIwMmM= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WdufjRKj; spf=pass (imf18.hostedemail.com: domain of kmanaouil.dev@gmail.com designates 209.85.208.177 as permitted sender) smtp.mailfrom=kmanaouil.dev@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721234016; a=rsa-sha256; cv=none; b=SiVuLoHU/8wSbLj7OxFs1c8IjY1yEVtUQutKiepV4XPZzLzXTFJBhAISPVGVJeghkLEKj5 SSKrldahKh9BYUojGT8t0A5sRAh1Rebu+oph6tNL9FaqtIF8ffev4LNN8K/DsbBvQQ32lp NoXnE+a1+t9uhP0EskW4VuAC8qDjuHY= Received: by mail-lj1-f177.google.com with SMTP id 38308e7fff4ca-2ee9174c5c0so8519051fa.1 for ; Wed, 17 Jul 2024 09:34:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721234045; x=1721838845; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=32qbVbdcIIo082PD1NEo17pI9Fe6+jCMt4sWhJ+oEKQ=; b=WdufjRKj8QhGsGp8USBgE+DrMZdYPn0mrnK+adQV/331Cj8+eh+eYr2e9/ir3mlJT/ LXPd0bqdLZRSMjnsIosZE07Q1pd9pc5NtlN9QrTgQxCh1XoRqHG2Sy7TBZ/She4F8uaR YVqrRoisoFRPhrm1o+dLHNVL1/HyMBDvxxbjH5+3FJF5XKYYqscOhTEMVOh2XEpJS057 BxLhIURwcnMaXYaSodb4AqQq3Nvce2JDcE2H1/L/RYuGkLhcYkUQl1Bdkbf8bH43QP+j xrU5b9KkHc48lEtUq2I+lgnBg2SkIBElQdHzD65wQwtIdwfmi0eLRSILdigw94GvCefk KQzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721234045; x=1721838845; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=32qbVbdcIIo082PD1NEo17pI9Fe6+jCMt4sWhJ+oEKQ=; b=SnKihSjZvkV/jqpm9E7M6/uzc+QYbTpgDUp+1mR64JodXCtrrVo9WCo1f/0u85B3EW qthCcpQBUXRhrymiUhlQHU+LeayOObPOpEgW8N1qG6TbPkKKonQGAarKlEZMNFtUPLCy +XBaRNzD/rw6oRITGZtjZhJR5VPZWUCj3ycGkswt/npAZZD3T165/FPkaixUH7mRPlVr 7pB83wKuArHmqELAsumX3y4HZrZ53vqjQKHliuFxv7yhAuRwWuzGgGaDgGlxGcOndk5L NTiBZQhNI2Iqef8EFTlGe7q1110oNcaKDIUWnss4Lay0zqpBSK9X+rUUDuJM00o+cU8e qJQw== X-Forwarded-Encrypted: i=1; AJvYcCWozOWIRaIbIBHBmccFkGHOsb3ebkgftkxpkeoiNbeHJiCYxfC4Z4yTkbid+Fw3L+8C0qk0SGyn7QBuxACGxFVfHhE= X-Gm-Message-State: AOJu0YzTG09+WzYuOdPZ21VuTe5AW0AnFJLwgcwj3NfuWAQhOyGQLwdG y9LKYhxwBBTYP07LIEEFbKXhRsN0af78BpOl9P6Ci5ANcI/S10Wy X-Google-Smtp-Source: AGHT+IFHoEqW03F6S4z6swFivCqKl+77KnSoWGbGJ5TX/ZETCILoAnLRjscrZ9ghdPSFooND/k9L8g== X-Received: by 2002:a2e:be24:0:b0:2ef:81:aa5 with SMTP id 38308e7fff4ca-2ef00810afdmr9009701fa.8.1721234045118; Wed, 17 Jul 2024 09:34:05 -0700 (PDT) Received: from ed.ac.uk ([2001:630:3c1:90:1614:6de0:61c7:40b0]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-427c77a5804sm3671585e9.16.2024.07.17.09.34.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 09:34:04 -0700 (PDT) Date: Wed, 17 Jul 2024 17:34:03 +0100 From: Karim Manaouil To: Vlastimil Babka Cc: Bharata B Rao , linux-mm@kvack.org, linux-kernel@vger.kernel.org, nikunj@amd.com, "Upadhyay, Neeraj" , Andrew Morton , David Hildenbrand , willy@infradead.org, yuzhao@google.com, kinseyho@google.com, Mel Gorman , Mateusz Guzik Subject: Re: Hard and soft lockups with FIO and LTP runs on a large system Message-ID: References: <3128c3c0-ede2-4930-a841-a1da56e797d7@suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3128c3c0-ede2-4930-a841-a1da56e797d7@suse.cz> X-Stat-Signature: azdnecfop11iahh5ddja5azo4fickzaf X-Rspam-User: X-Rspamd-Queue-Id: 198CD1C0025 X-Rspamd-Server: rspam02 X-HE-Tag: 1721234046-633171 X-HE-Meta: U2FsdGVkX1+CqtrTr7jVB8ytSc2uOfJzRZpUGo2Vb5iBH6lilmz6UGFqQ7d4ZdtPDV04Br8gbDrMduxA/FTIAxThHbEQjpFqDk1UOhYXRbuqb3JmR1ZLJyh2hnsgS6DwRGvFr5dJs2yPyQquYwDIj5pWDOTuNk+P5EINuWUWF4W17sJFMfvss6RzIPjfXc7exLSHRbJ8ks1IWDpnZQwKANZAMahM251jwjoUUlnJGZYR7DQCTbRgI2owxzR/XFQ9WHWTttyWylldC4g/uJwbALrodgKh6LvztBBM9tkqhRl/AvjIsKF4z+rmBDBOkZdYWS7UOVdB1hk/kDRypq0/c1fZYs8VRYuM0WFLlIGTi8YEaTyBLJ3vQmfRXknJsXiK1ilCJtcXEQFAm3UltT9QKyfTGsgBmnObNN/vRZQ5uKHjc3PC/J5+yhMbltuF/X5JzaJ0Wi1dLucMVm/C+zzcIkpZKJj1dm/TGrWo1jzOujyym0mHNxL6oGzGXa8fD604XUDpOUEiOquepi08lTDjYm0D3EfbddRqy7bIZt1Ry30to/kyazXHIlAEvqQMkMuI0cVM6CSn1DWa5ZLzyrebUc467hD1Hj4+9AJAigPoCWLBP5VJd587klSR8oBJyvl718ZqfrD413K6VB6mnxB9V8+Mb9ADbusKF9ytxyaEJOs29uPNnFp5WhyrrY9Hf+gbTv1B7snpxALenDmegUirXapuJCQQFU3LZA3my/GsVXkobYqT+EtJJzVfD2YjGOhl0JFfpBuPq+YSUm1Imiodf2emH1hoiu+LGVMiPqkz6s03gVFLiq62YhI3qrTM6yvrovIV9sneK4SdbFG1fdRPaAJpRhNfzRibGRC5igCWRNg+40RPXHq0ldnlg0jULM0IuB/JfekNWwoh5FBBu0TdVuhbO/gVPnCQ6pZvP00yfHcnQqVavflo+M31s4AvyAwvg4qSIHjVpkmazyUaOt+ wVWOaOqi +M4BV4khPNzpHJ2lEAwInlwc8pZlCCx+kY8xHcXsRueIuzFxqywlH+8uKQqwav8r8MVDQi5KdyRfr8LvHIoi7Bp6nCqS9hiYUM1EYg76gUUCJcs8YUPZruQwkENS9m9W9UtmLAs3Nhqx/ottNmOqmcekixniVfAFRkLEDCiOOqGZydJJCM0RSgg3lwYXpaR+z1GZCUFA940Tkm5aykem1cRVW3askWbFDiOYZyK4qZzJQwk8/3WB1dGXqfi+9YqFjiQ5ELHCKys7NXVziZNOdxJytgh3ZAsWO1C5vEjck2mblqxn9Lj5YvIlIQX1nJ7UHoR5+aIDr6370dk0Ci93SE+7/xHiO5IBj7eT/1p0VZglW3rgdCcUGmnPfCbJtsENzkIpm3c3ZNnvkhCHS+7kYfud/O1hu6CrasLeg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000010, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jul 17, 2024 at 11:42:31AM +0200, Vlastimil Babka wrote: > Seems to me it could be (except that ZONE_DMA corner case) a general > scalability issue in that you tweak some part of the kernel and the > contention moves elsewhere. At least in MM we have per-node locks so this > means 256 CPUs per lock? It used to be that there were not that many > (cores/threads) per a physical CPU and its NUMA node, so many cpus would > mean also more NUMA nodes where the locks contention would distribute among > them. I think you could try fakenuma to create these nodes artificially and > see if it helps for the MM part. But if the contention moves to e.g. an > inode lock, I'm not sure what to do about that then. AMD EPYC BIOSes have an option called NPS (Nodes Per Socket) that can be set to 1, 2, 4 or 8 and that divides the system up into the chosen number of NUMA nodes. Karim PhD Student Edinburgh University