From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1344DC54FB3 for ; Mon, 26 May 2025 21:09:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A9A96B0082; Mon, 26 May 2025 17:09:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 65A336B0083; Mon, 26 May 2025 17:09:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 54A956B0085; Mon, 26 May 2025 17:09:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 335E06B0082 for ; Mon, 26 May 2025 17:09:45 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id CED88140E47 for ; Mon, 26 May 2025 21:09:43 +0000 (UTC) X-FDA: 83486300646.18.1929709 Received: from tor.source.kernel.org (tor.source.kernel.org [172.105.4.254]) by imf02.hostedemail.com (Postfix) with ESMTP id 4D7CF80003 for ; Mon, 26 May 2025 21:09:42 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=STloriLV; spf=pass (imf02.hostedemail.com: domain of sj@kernel.org designates 172.105.4.254 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1748293782; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=Gvw0T/DRKISqFz5/4r/Au2mJ0rdFD/LCdriemMyfMVk=; b=kNxOyoMC0gRGHufKzmvPLckADkVBunik0faI+hnRSfTi2oEahF2ElV4lKmdDNHXY4ugVKS guaYfqHI5h90S3xPTY9knjlunKcsOYpOlHgQ6XsGOmkqXYaDk19q3k/2SQRNojVZCNG2vc fJDaU8kIR67GEoX6bPQl0TXc1wsIwsQ= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=STloriLV; spf=pass (imf02.hostedemail.com: domain of sj@kernel.org designates 172.105.4.254 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1748293782; a=rsa-sha256; cv=none; b=H+qFTidLOBIKlYRLwy7Wad4fNTNiNDuyHyr4cJXVm3uDWD/NaKspbcaGMyf0NwKGsEHNPe /ocQoqD/MFZi/L1XdHVDfQw5+TZIa5YFjOBzPXp2WjM3G7lQ7YEshahokwZ7HnpFUxiHP5 g3A5k2RH7mk9WbsVyw7doS3MzSSTTLE= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by tor.source.kernel.org (Postfix) with ESMTP id 54555629C8; Mon, 26 May 2025 21:09:41 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D8D41C4CEE7; Mon, 26 May 2025 21:09:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1748293781; bh=Oi6m9zluj9+upljpAJCCZuQWC64rKRxq5v43Dw2FA6s=; h=From:To:Cc:Subject:Date:From; b=STloriLVBeMKppjjdQPNQViDlg9IzHsHKwKh2FvRoKdtK3pMZR64tmTyNr8jTPRra mwCXo2djmbcXwFh3Diw76oAiHdM8luheSHGssJBt4mCXSZuhnVM978uW7xhnHz8tvx k/8+GT71v+R+mvZ80fxhf2sTgDemhQGb9pBOiKNUGaZKkqp0ycQm9dbzsuDruAdqtK Wsm9DyEsoyc9H1trgTSrpi+1mxp8Ntrbk0DtVWQHBQK2PIOBTRwKOqSvXPVyTeAmZw i82rZkmZ51I8D60XTxj45vPJd3t98e+ybS2V42uR4CVw2thERJGSO8sA080/rx/wBe M/rKecMlArLyA== From: SeongJae Park To: Andrew Morton Cc: SeongJae Park , Jonathan Corbet , damon@lists.linux.dev, kernel-team@meta.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 0/4] mm/damon: introduce DAMON_STAT for simple and practical access monitoring Date: Mon, 26 May 2025 14:09:32 -0700 Message-Id: <20250526210936.2744-1-sj@kernel.org> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: 6jysoc6x5i4tk449iy33zg7wbw3mgite X-Rspamd-Queue-Id: 4D7CF80003 X-Rspam-User: X-Rspamd-Server: rspam02 X-HE-Tag: 1748293782-495230 X-HE-Meta: U2FsdGVkX1+vdjd5ZCuUY/Xbt3vYHnIus0SXcIyFfkkYL2EgchjMux5ifZSmRRE7jAdGpJzJ/PxDy41Tdah/yZtlEH8clsznUA0zuUjdXBhvIVGJMkRgiiiBxqf3idxmDWoo31h6uTBaOfG7APlYsILovkwm94wLZ+3nOO7Rza6VXqNnbFDQD/wPs4TaszIqim0zhoV54gd/EecupKXTkdCElPIUnFJpXL5ubgRIljILzTOheUaR3cokjQVKT8UKzZGF4s1KJUdoq13WMoaCkoSmNR88fcPtCg/ADcc5Wae3I6gERMT5dAjAtDpbPuAnZjc97U/F74bZor4X0P1ESE70K8IXvWxlwZCJK9sO76FtKarUdEHLS2bZ//VjwI2/KCBocUVYkySygdC2WH30iCudHAYsrwU89nwjbPmPzFdXmkmU4cBNvDLqytS5Pilz2PjggrKSiu8Ngx0sKkE+5FKN1QIHwIb356WtWI8ONgpD6IwkXXEzHykOBSMLBkjvgztDFzhuPMtbqToug7rWzZ095U2Gpms5/D9lIOEcqN0y0Y08V84D70uMFEdpjsZYh8WyNGxDaBF5q0HRg5iYReNLKYk/yGNzVJvplSUAmFzGf/vFKCTFWsxGblm1rs3JhI0lsvHwDu7nJwykTD5lmd95kUHVctmlEwwi40WyROZ83aZ/qHh07qOgzivCL6JHW6q+iMTncj2gWwebp/hQh9RrTj4d9ct1H2dNYg51SXeGcydHitbs7ws8dNpSONEqXIGVumgK4A88KVZHEiHZQpr0kfz+Ji9LvXeZyCpMotZ+AaVvuEbFbSTGuPoVDuciOEfEdqyYrv7KVuserdiqzPrRcXlAbjj1sPRyx7Ck7RCLvrsbt6rQ1HrvLDxoyN0mXbJP6v3c1kto2MccgC7VDZCtUSgma09+YmAj5x/d+nubOei+mdPog3BAZmA3XGJg0xUvDfMjJPDc21/WHTK 4fDa9EaZ WYnDnLudJOBKkf66lHUvQP/pW8uFKbKQ0uIEZoRkkotZ9hlQRt9XiiIt733BPr5Ul8MXDvymhw20N7ZNmlH+rj/IwYFn+/vrg4evkZVEHDodY4iLmwZtCAzzJYPeYDWLXadV1arwEkJ/xvKQPlq04IFv3xqP7J5wpV0Gb+7/WB4sMfxNksEPQ2aD9bOl0qnQVgMmIkbS/G0ZMKugRK7WJdhPSWzFBdw0KPXFXkVj+L5GFfcJDwGxAlmEZjMEAlaa1oFiRCEWfZbeMk94uTdNs+MkLLw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: DAMON-based access monitoring is not simple due to required DAMON control and results visualizations. Introduce a static kernel module for making it simple. The module can be enabled without manual setup and provides access pattern metrics that easy to fetch and understand the practical access pattern information, namely estimated memory bandwidth and memory idle time percentiles. Background and Problems ======================= DAMON can be used for monitoring data access patterns of the system and workloads. Specifically, users can start DAMON to monitor access events on specific address space with fine controls including address ranges to monitor and time intervals between samplings and aggregations. The resulting access information snapshot contains access frequency (nr_accesses) and how long the frequency was kept (age) for each byte. The monitoring usage is not simple and practical enough for production usage. Users should first start DAMON with a number of parameters, and wait until DAMON's monitoring results capture a reasonable amount of the time data (age). In production, such manual start and wait is impractical to capture useful information from a high number of machines in a timely manner. The monitoring result is also too detailed to be used on production environments. The raw results are hard to be aggregated and/or compared for production environments having a large scale of time, space and machines fleet. Users have to implement and use their own automation of DAMON control and results processing. It is repetitive and challenging since there is no good reference or guideline for such automation. Solution: DAMON_STAT ==================== Implement such automation in kernel space as a static kernel module, namely DAMON_STAT. It can be enabled at build, boot, or run time via its build configuration or module parameter. It monitors the entire physical address space with monitoring intervals that auto-tuned for a reasonable amount of access observations and minimum overhead. It converts the raw monitoring results into simpler metrics that can easily be aggregated and compared, namely estimated memory bandwidth and idle time percentiles. Refer to the commit messages of the second and the third patches of this patch series for more details about the metrics. Discussions =========== The module aims to be useful on production environments constructed with a large number of machines that run a long time. The auto-tuned monitoring intervals ensure a reasonable quality of the outputs. The auto-tuning also ensures its overhead be reasonable and low enough to be enabled always on the production. The simplified monitoring results metrics can be useful for showing both coldness (idle time percentiles) and hotness (memory bandwidth) of the system's access pattern. We expect the information can be useful for assessing system memory utilization and inspiring optimizations or investigations on both kernel and user space memory management logics for large scale fleets. We hence expect the module is good enough to be just used in most environments. For special cases that require a custom access monitoring automation, users will still benefit by using DAMON_STAT as a reference or a guideline for their specialized automation. Revision History ================ Changes from RFC (https://lore.kernel.org/20250519164415.43935-1-sj@kernel.org) - Add an admin-guide documentation - Wordsmith commit messages - Rebase to latest mm-new SeongJae Park (4): mm/damon: introduce DAMON_STAT module mm/damon/stat: calculate and expose estimated memory bandwidth mm/damon/stat: calculate and expose idle time percentiles Docs/admin-guide/mm/damon: add DAMON_STAT usage document Documentation/admin-guide/mm/damon/index.rst | 1 + Documentation/admin-guide/mm/damon/stat.rst | 69 ++++++ mm/damon/Kconfig | 16 ++ mm/damon/Makefile | 1 + mm/damon/stat.c | 245 +++++++++++++++++++ 5 files changed, 332 insertions(+) create mode 100644 Documentation/admin-guide/mm/damon/stat.rst create mode 100644 mm/damon/stat.c base-commit: 90887f57d7a67917136e7c70d26fb3f2fcdc6f53 -- 2.39.5