From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA39CC5B543 for ; Wed, 4 Jun 2025 13:22:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7A4EE6B05E9; Wed, 4 Jun 2025 09:22:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 755526B05EB; Wed, 4 Jun 2025 09:22:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61DC66B05EC; Wed, 4 Jun 2025 09:22:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 42DB76B05E9 for ; Wed, 4 Jun 2025 09:22:34 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 8DEF1BF06A for ; Wed, 4 Jun 2025 13:22:33 +0000 (UTC) X-FDA: 83517782586.11.3A10AB4 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf25.hostedemail.com (Postfix) with ESMTP id 0EF4CA0007 for ; Wed, 4 Jun 2025 13:22:30 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=Wd+QjN0T; spf=pass (imf25.hostedemail.com: domain of donettom@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=donettom@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1749043351; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AyofCP+ZN+6ircaf7mXhdeuKJnDTj+s9f0clMqZxlXs=; b=Ggd4HmH2owMtyTwogLkfoAcSl8PBqgK5ppnC+elaMeDp4bu1qOZhAqgLJ7RhPxzsqvkHAi SAtzLbpBjGEq4XVtGD5UYgI3Nn78de5lPzUqLTH9hK29Ju4oCIVxKrUhxmtur3lQuG7ggJ bhNMQPRtk1JAILqULDAUlAP++lcS6MY= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=Wd+QjN0T; spf=pass (imf25.hostedemail.com: domain of donettom@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=donettom@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1749043351; a=rsa-sha256; cv=none; b=YPBYQQet/n6+C1pbb7Ieem+TxFi+J/n26rdvbKJggzopMr+qDi8QtbR2Of4oc/AOxbOqFL woBt+R5G7aFhYV6Nkb/2oP/ug6IQ3mVIrMpwgwg1x4QOY8xYlTWI+5zqB5q4U3UDrvqqaK jM+NCmMKRscvDz5GM7sA1yKRF7vpLE0= Received: from pps.filterd (m0356516.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 554DCawf032172; Wed, 4 Jun 2025 13:17:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pp1; bh=AyofCP +ZN+6ircaf7mXhdeuKJnDTj+s9f0clMqZxlXs=; b=Wd+QjN0T53cJ5UBS0BP1xh UoNltEcgex7nx4RL2dI/9ob1j+9I0sh8lGb06Z8yDECNrMDf1LZWEPMwMv6I8s5j IcNdrV/mfCpCobGpLaKXpuAyhzs3gga7PjSFbyY7C+OBpr6RSVsBGJ6PegI0tRjD lm0JhgdAQFrFiZhLqssqlyD53tZa1D9qQZBVRqkatFL6+hiF0T8PwA4u+tx31VC1 buUgd7LyC28IYlSDhDoqBHJvUXnJf8cHwtRBUch7NcdoZ0YMNu+awjzOxrfpk6nm WvELcIcPxBe6adYvR3VwJBbi5Lppb+StgHvIFoYyGY3X0WVB02NDLSrsBnl9r75A == Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 471geytmnc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 04 Jun 2025 13:17:11 +0000 (GMT) Received: from m0356516.ppops.net (m0356516.ppops.net [127.0.0.1]) by pps.reinject (8.18.0.8/8.18.0.8) with ESMTP id 554CxI1B029221; Wed, 4 Jun 2025 13:17:11 GMT Received: from ppma22.wdc07v.mail.ibm.com (5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 471geytmna-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 04 Jun 2025 13:17:11 +0000 (GMT) Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1]) by ppma22.wdc07v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 554BoRl0031650; Wed, 4 Jun 2025 13:17:10 GMT Received: from smtprelay02.wdc07v.mail.ibm.com ([172.16.1.69]) by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 470cfyyymw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 04 Jun 2025 13:17:10 +0000 Received: from smtpav05.dal12v.mail.ibm.com (smtpav05.dal12v.mail.ibm.com [10.241.53.104]) by smtprelay02.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 554DH8MM25297572 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 4 Jun 2025 13:17:09 GMT Received: from smtpav05.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AD4BB58067; Wed, 4 Jun 2025 13:17:08 +0000 (GMT) Received: from smtpav05.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id EF0ED5805D; Wed, 4 Jun 2025 13:17:02 +0000 (GMT) Received: from [9.39.21.166] (unknown [9.39.21.166]) by smtpav05.dal12v.mail.ibm.com (Postfix) with ESMTP; Wed, 4 Jun 2025 13:17:02 +0000 (GMT) Message-ID: <9f7ae0e6-4640-418d-a4db-dba594377ac2@linux.ibm.com> Date: Wed, 4 Jun 2025 18:47:01 +0530 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v7 1/5] drivers/base/node: Optimize memory block registration to reduce boot time To: David Hildenbrand , Andrew Morton Cc: Mike Rapoport , Oscar Salvador , Zi Yan , Greg Kroah-Hartman , Ritesh Harjani , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Rafael J . Wysocki" , Danilo Krummrich , Jonathan Cameron , Alison Schofield , Yury Norov , Dave Jiang , Madhavan Srinivasan , Nilay Shroff , linuxppc-dev@lists.ozlabs.org References: <2a0a05c2dffc62a742bf1dd030098be4ce99be28.1748452241.git.donettom@linux.ibm.com> <20250603200729.b7581e017e4ca63f502c795e@linux-foundation.org> Content-Language: en-US From: Donet Tom In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Proofpoint-GUID: 4FsplurUS7c-nNdN0AmIoqUxedlpBq1v X-Proofpoint-ORIG-GUID: cm-Oc-tEKoBA3qeJIzbOxbnJXz9fYfvv X-Authority-Analysis: v=2.4 cv=X4dSKHTe c=1 sm=1 tr=0 ts=68404757 cx=c_pps a=5BHTudwdYE3Te8bg5FgnPg==:117 a=5BHTudwdYE3Te8bg5FgnPg==:17 a=IkcTkHD0fZMA:10 a=6IFa9wvqVegA:10 a=VnNF1IyMAAAA:8 a=k8bw5oIpvhWMPjh-KD8A:9 a=3ZKOabzyN94A:10 a=QEXdDO2ut3YA:10 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwNjA0MDA5OSBTYWx0ZWRfXwW6CTbwXdFug zCXnZycTe/FqRgU6/7hAGmicwbUedmIIArWprAYdtHHYFavfFa4G+KxKc+VYO1pl6alcHRcrnEf xtqqWsw7o4wouJXJZzoiVu80TLtzFLtJwUNLpc6hsMSpgi2KAPuhJs5p3mGD9U84OeoklTUc5sn PO0Oz36/aec/Qkgp/dwSynzy9BzkcF+qgNlnrvE9Xo4/YUm9T/pCxhQS/mkX2twpZ2oBIWO+y4E fLsZmb8Hm2OFz2Owb8eVOZs0ET6iz52pippiIXA9bORtj7zfZU5Vif2yLNxoL/mevj+FhZvouV4 jeOJ8otynWo0iTOxhMKjLDCuX5C0uN78khP7zH28mqiv/t1+dsNkBy3020SYjol1DSCiNLzXe9j DxyfRX481RrnuSrz9kupxaY++cA4LzgAC6gDwjUeAhnBviMJhEI4V4hGPkwPy4jkwFCF2rTS X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.0.736,FMLib:17.12.80.40 definitions=2025-06-04_03,2025-06-03_02,2025-03-28_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 priorityscore=1501 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 clxscore=1015 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 classifier=spam authscore=0 authtc=n/a authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2505280000 definitions=main-2506040099 X-Rspam-User: X-Rspamd-Queue-Id: 0EF4CA0007 X-Rspamd-Server: rspam09 X-Stat-Signature: xefxpjdbnt985bfwz3hrngrk5n69d41d X-HE-Tag: 1749043350-847995 X-HE-Meta: U2FsdGVkX18IRXNtfsQMubzZE613ETSOTOUArHqUSKODRVitUV+Fx+wGnU1ZEOSLwsIKkyol5G1kJ8sAmKbgqcWkSWZmdAtzj2I9MeH3ea6jfi8Jlm9k57JnOkT+JLyyxo4XkaX9nvB6hsvUhYXr/lwAmLcAGe28rPJLGAYpVcu5q8Js/2lZFCVl30fczw7db6ZPA4hQ9H4yqavKVigyOu7YMgBCYdGJ4KuVVUsXevfqCplXLSD8Yn+tmP13Rc/4uHcXd73+jRLW7Lwxh4emM3pgrQOzFk3+t6Xi9h3xULB0iFRwiEotq3641w7vPcrrfqahpXPmCn/4uummZ00J1fWkKl1dRrCHa4AxSR6NCKFwOxJQCcGvqyyN+48kZmNKMzKE6ork+qkeUspgXFOupbK3KZHIO53K0WxUGC9CEx+jKK8isEP1Y+qliHIraayW3LcWf2IRCjwbFnZpGzxhDzhMkUp9ri9L7mY8qJCUWmGDH18kpnwNFGEJpkr9AxQaYoiul/OkBp4WMuplQRM05HmpjIogRGVvNSjzju7GcGLVy1PYih4r22/8+zmgfsW6N+sFwWOtckFaeJ+bz12XSc463WyVrUqo/nCsWnG3M6xMW3RND9DbZ9UXUyxskMVqFyrItOOHBKdhZD4Zn0qWTNHVBBoHNHlHnEzOdPD2lSYYPtHrWEQEx7ml5AqiLg+3TXNK09THs1qTC7iQoGU1MPUZfcTmSOAuH6jRFK32uDXgeggPjq0HNNi8Gb2wB1SR0WPLyyOxpGH2LTYERlanD36oj4bbh9RY8xuwYEuugapLIJyJUXh846go29w0QAEkNYAsoqUoDUNFMjMeNS/GEMAlsskVRn4/JARU6pAGAB1awiVCUcGeuV4GmftCloCOzvwJUVBRmn3DVBMtFZw7azieUexx5WJUgiO+VCCy2SOwNMTMdFgbR8WQbzet2Eq1aCm4L0o+WLBvOdUhonT 7afX3M+M sfpD3FuNCrwi+jhMyuPity+It45H3OJegpLddPkEJR15DZ0/NycUNDATOedJVn7NsbeRJlaZ/1aW5untUh7GsrPICf5KKG53KmWSRpxyggKcna7F8K/wfIGtANSeOCI+qoCC2OdJL1s5SMhy0I6/9BFrWbLKY5c+Zu68X10z25JhLnTdVAyzqCRydXStwRAxZDD3bcmM4biy+0/B1Uri2N50OtNweRy2mmao4hFVcYoq+WSz5fWqRDWmbmzIS9SpcO4wYS0opGUbIisngX25EDYNffJiBmKShS9Upn0ZxOE5s0L/lJcGURSGO5wpSciRpeUZfhmvroR4O4HA7dzUDhYd1WKbPwy5/o+Te X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 6/4/25 3:15 PM, David Hildenbrand wrote: > On 04.06.25 05:07, Andrew Morton wrote: >> On Wed, 28 May 2025 12:18:00 -0500 Donet Tom >> wrote: >> >>> During node device initialization, `memory blocks` are registered under >>> each NUMA node. The `memory blocks` to be registered are identified >>> using >>> the node’s start and end PFNs, which are obtained from the node's >>> pg_data >> >> It's quite unconventional to omit the [0/N] changelog.  This omission >> somewhat messed up my processes so I added a one-liner to this. >> > > Yeah, I was assuming that I simply did not get cc'ed on the cover > letter, but there is actually none. > > Donet please add that in the future. git can do this using > --cover-letter. Sure, I will add cover letter in next revision. > >>> >>> ... >>> >>> Test Results on My system with 32TB RAM >>> ======================================= >>> 1. Boot time with CONFIG_DEFERRED_STRUCT_PAGE_INIT enabled. >>> >>> Without this patch >>> ------------------ >>> Startup finished in 1min 16.528s (kernel) >>> >>> With this patch >>> --------------- >>> Startup finished in 17.236s (kernel) - 78% Improvement >> >> Well someone is in for a nice surprise. >> >>> 2. Boot time with CONFIG_DEFERRED_STRUCT_PAGE_INIT disabled. >>> >>> Without this patch >>> ------------------ >>> Startup finished in 28.320s (kernel) >> >> what.  CONFIG_DEFERRED_STRUCT_PAGE_INIT is supposed to make bootup >> faster. > > Right, that's weird. Especially that it is still slower after these > changes. > > CONFIG_DEFERRED_STRUCT_PAGE_INIT should be initializing in parallel > which ... should be faster. > > @Donet, how many CPUs and nodes does your system have? Can you > identify what is taking longer than without > CONFIG_DEFERRED_STRUCT_PAGE_INIT? My system has, CPU      - 1528 Node     - 16 Memory - 31TB I ran the same test with and without CONFIG_DEFERRED_STRUCT_PAGE_INIT, and the boot time was consistently higher with CONFIG_DEFERRED_STRUCT_PAGE_INIT enabled. I'm still investigating this. I'll check further and get back to you. CONFIG_DEFERRED_STRUCT_PAGE_INIT enabled ---------------------------------------------------------------------- 1. Startup finished in 12.959s (kernel) 2. Startup finished in 13.036s (kernel) 3. Startup finished in 12.944s (kernel) CONFIG_DEFERRED_STRUCT_PAGE_INIT disabled ----------------------------------------------------------------------- 1. Startup finished in 12.234s (kernel) 2. Startup finished in 12.287s (kernel) 3. Startup finished in 12.230s (kernel) Thanks Donet