From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C254C61DA4 for ; Sun, 19 Feb 2023 18:09:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3ECFC6B0072; Sun, 19 Feb 2023 13:09:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 39CCE6B0073; Sun, 19 Feb 2023 13:09:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 264676B0074; Sun, 19 Feb 2023 13:09:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 11EF86B0072 for ; Sun, 19 Feb 2023 13:09:24 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id C83EDC07B3 for ; Sun, 19 Feb 2023 18:09:23 +0000 (UTC) X-FDA: 80484828606.05.BCA4105 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf11.hostedemail.com (Postfix) with ESMTP id 31FF240010 for ; Sun, 19 Feb 2023 18:09:22 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=rXhkJ0vv; spf=pass (imf11.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676830162; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=r0tcqI0j7aEgUeZ3qfqqSLjM6AuYGbBtOD11BD17v1Q=; b=HlGHHAuwpXkmMmiMh9OYRMJGP67mrzL2c4qturGgRdaRSWzG7mSZKpbru32ncMH0LsBKGT bgZAUkWDZ83AxJnzxW8WCQZV/WhHLzELHHiNG359Sx03KF3gOXJMHRoa4CDcQRyc0XbjGI kYKlZmL37p3MEHs4BU6dqs3V3cIzEWs= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=rXhkJ0vv; spf=pass (imf11.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676830162; a=rsa-sha256; cv=none; b=8FeoUjmh3fpIb8Y47jF3tEozUS2MmcuDv+AWvdE1zh9yR9b82g53zk+jIC/B7IJdRGQNoZ 5uHqx8bqjYMzIa4K8bQSABPTdI1RnvNS75crnWk49rSFUCC6xeLtjdf/2qVcFO1qVuoXL/ 2nOHwGnkxGFdG1ZllGgrZY89EKaf0WI= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 14C9260C08; Sun, 19 Feb 2023 18:09:21 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7E7C4C433EF; Sun, 19 Feb 2023 18:09:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1676830160; bh=Pu+Gp8T3EhZVIEzsjwHbcywcdU6U9H3ieQrvba/gz6g=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=rXhkJ0vvOS3Je55Nj0PtCKHYhqNKxgtVYw6Y+vGMOQWNDICniBZGvGtLwizrDfc8i f8ZLB4X4rM61bgBDM+wXMHUmxsdW2ibBP4DP/E8cmK2d3QgCZ12Obuw0o/kg1ka8ya mY/H0vd0cU6H84MDZ+Il2ziaqctBdNI/ApNkX4kCfSio9JJd+8YHAs405SmzqgzGrO aRA0Exl/Mb8eVTOKFp2d88hdwWUHD3aWJp8ZdwHAlaXFN5sCDHwgT2EUl8YdzL8TpZ sl+5PW6KinZWYx8fbdicJ2eQb+pvnjhJPuIX59Tb8JZemU6eCm0r9Qq4fn5XRseRJa eaR6VtxKMJDwA== Date: Sun, 19 Feb 2023 20:09:07 +0200 From: Mike Rapoport To: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, Aaron Lu , "Kirill A. Shutemov" Subject: Re: [LSF/MM/BPF TOPIC] reducing direct map fragmentation Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: tdecf3cgmi8rkrkum63nzt8d9i1ma45f X-Rspamd-Queue-Id: 31FF240010 X-HE-Tag: 1676830161-519361 X-HE-Meta: U2FsdGVkX1/QEdt3ziOMlMuuq0HT/PJrc9JS03MbRMsALUhcnT+nzo4ucu6W1z5Q30ZEtuGhiYLGvNVTohFEu+b/BuBl/yw0MXMZk2yb5a2K61uNpBeTiZGDdySIadmO4kZUdluUeEGKQW09M2ASWJPLhv+HE19P1oJwWEXk3zq0ZyhUVenpSQnWxc+KgUhxv3pii2SFDWsFAvliX3asIBJk6uliO+QyAFLylj1wDVw6u8I4z2/dGgkcXj7th/FukVWX+CBm3KjR/ZY/6gwT5b6MOBLc2l11PCfrDCjmqEkgzklgKO/jXcPHWIU4C7VKKVDDuZjINVlbfwzEt0ul/PhpzeFCfNzH1C3aK+EbggqLeRCT0qvkUKMTt1OOT4P925xRCDoZmmgBNv3x3Ata5gxJAwYOH3Sl7ZETq6xtAek7YXVpaK3IpGo9ZfgIqwbrKEeZcwZuC5z0tIY95ttqfSvRxsunteLImCA17L5AlBxJwX9RuuVZEZglNsBovr8XRosksPI6QlX405tuI6lR2SRW68BJEg6SJzAV43AthopdA9LfCuoEAoWTKKzYoC5+q5TMQhZhzt2rKvIPC5E+uju5YQ5NijjJeueft9X6xceRb7fxrQ2zrxWq+Eow4fRkI0y2sCY3LMc7b3bSTEdkem1fhnIRw+kWCkXIxef6EC23CSeWS3icnH5FOmpzkJknjxrb87ZiGJUB07/oEbc5hAwo3BAMbwRgMduHaa0TOqIOfMvYo4LIQoSw4TXRJNHevqWdTAr5MJOZl7XuWSPUkBIPfStGbRqYyy14SeotZxfRgXO+UtyTzAa89A6TdPULMJf1wTq/BgaXRqki3UQQkvOQHcpGp+wqZX/TVNE71eEHbUMRB3vKFQw/mSr3m9btLDALPo0sTjyXOVpBkB9O48p8B3FxQ9h9AEMQ4U0H+KkEvT6NV/6I6utN8erx69bmmbu9w4XiWQ8uV3UDSVM eK5/buoa wQJT/E5OnNOH+p87v9zI02AOXITMdarCZelV1qYctZxOeCIphlFbNFhnzEnkpXMoQ99POG+M3hhKhzP+vt39N6Z5YFOX04SszPQhILZRAz3W5GFyhYKT9641mYb+hVGuglKB7I2zn4B2CaqHvRu0vJSevzlel4xnc6PeyEFntWuqtDctdDj1xeFXJ/91s0lCqWXqc+7G1iH2HZPfrCcxpmKSVmlLcoj/974Ra2r1UsA9dywy9DjVOUUZk29Vq9sDNT7pJ8HzjkzvT6Ln+fAnh5p+8H4gJq5vFslUVkGDgs/qmr/3iuYDZkU4+W/wy6EfTjUcOOuh6D9HfoPrfiWdi6ArPltDVHzQl7pl+wK2bLds3V0GmK/ysHn4MwUpkU9z5wsT1/8ncJcXmVGm11r2d4FgPb8zoEhkI7hxG1e4WmwDt87s1fEY7trdGp031nEf5I89TfXIBW3r6uOpd6gNQUZBUcIqMyFJlpU79pPx34BL2oG8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Sun, Feb 19, 2023 at 08:07:59AM +0000, Hyeonggon Yoo wrote: > On Wed, Feb 01, 2023 at 08:06:37PM +0200, Mike Rapoport wrote: > > Hi all, > > Hi Mike, I'm interested in this topic and hope to discuss this with you > at LSF/MM/BPF. > > > To reduce the performance hit caused by the fragmentation of the direct > > map, it makes sense to group and/or cache the base pages removed from the > > direct map so that the most of base pages created during a split of a large > > page will be consumed by users requiring PTE level mappings. > > How much performance difference did you see in your test when direct > map was fragmented, or is there a way to check this difference? I did some benchmarks a while ago with the entire direct map forced to 2M or 4k pages. The results I had are here: https://docs.google.com/spreadsheets/d/1tdD-cu8e93vnfGsTFxZ5YdaEfs2E1GELlvWNOGkJV2U/edit?usp=sharing Intel folks did more comprehensive testing and their results are here: https://lore.kernel.org/linux-mm/213b4567-46ce-f116-9cdf-bbd0c884eb3c@linux.intel.com/ > > My current proposal is to have a cache of 2M pages close to the page > > allocator and use a GFP flag to make allocation request use that cache. On > > the free() path, the pages that are mapped at PTE level will be put into > > that cache. > > I would like to discuss not only having cache layer of pages but also how > direct map could be merged correctly and efficiently. > > I vaguely recall that Aaron Lu sent RFC series about this and Kirill A. > Shutemov's feedback was to batch merge operations. [1] > > Also a CPA API called by the cache layer that could merge fragmented > mappings would work for merging 4K pages to 2M [2], but won't work > for merging 2M mappings to 1G mappings. One possible way is to make CPA scan all PMDs in 1G page after merging a 2M page. Not sure how efficient would it be though. > At that time I didn't follow more discussions (e.g. execmem_alloc()) > Maybe I'm missing some points. > > [1] https://lore.kernel.org/linux-mm/20220809100408.rm6ofiewtty6rvcl@box > > [2] https://lore.kernel.org/linux-mm/YvfLxuflw2ctHFWF@kernel.org -- Sincerely yours, Mike.