From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09B4CC4167B for ; Sun, 3 Dec 2023 19:37:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E91F76B02AA; Sun, 3 Dec 2023 14:37:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E41F76B02AC; Sun, 3 Dec 2023 14:37:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D302B6B02AD; Sun, 3 Dec 2023 14:37:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C35E16B02AA for ; Sun, 3 Dec 2023 14:37:54 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 86C60A01E4 for ; Sun, 3 Dec 2023 19:37:54 +0000 (UTC) X-FDA: 81526517268.08.0D1D422 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by imf06.hostedemail.com (Postfix) with ESMTP id B099818001B for ; Sun, 3 Dec 2023 19:37:50 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=R5KWV1kB; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf06.hostedemail.com: domain of sj@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=sj@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701632271; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/G/aFrBk8WKDESS/zRzl9OXlXpzzqrQuOsLKTva2/NY=; b=pheTMOGVIP78/o3nqFquus+k1Du+43Mq7q39YIS2e/g4w8+bNn04H4JbHfJmrTdjbcFSz8 RVuIbxWb8yi3nU2+/yM0V01mUJAl0lJ4c91hv6Gd+6cj0lANN7yYDlcwEUoFq2dpTlB5Sq 21FOLEJBJtUm2QZcAgnEyHYj7JLzAz4= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=R5KWV1kB; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf06.hostedemail.com: domain of sj@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=sj@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701632271; a=rsa-sha256; cv=none; b=6qRtCrUOfIYNYQW2XJ+rJiWFXnZdEtvoVxOLY9uimnijr7nWCL0o5J0BrdKfxW8Mz+ojl4 LkIxuJ8OXQZ386yEgdLhPGRm+LwIDQ4PvwnMQnU92mCmCgRQ9lNB5OkT1VyETIgQiF5wCk T7OPRjEKdoJYh+JwM/ddJzu51RLYDQk= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by ams.source.kernel.org (Postfix) with ESMTP id D0987B80CAE; Sun, 3 Dec 2023 19:37:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9E697C433C8; Sun, 3 Dec 2023 19:37:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1701632268; bh=zpmUNVwx3eN/sMGL2XuHjExRiDp+GF7SBd2lpzktD3w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=R5KWV1kB8aQPhXpkQtSa2PKnVrkqiemTIiCSUaLmtURKjxX2D8lOeaa55ItuqOmHw khmxJIWzIw8Z6tewelPOqzOzL2NxBBjXSmVg3+eC7Q+eKJXijxq1xxP4AVNenxm0aM yZjF1BgbM6qoMKYhEv0xEk5wD7chUM1PQPqZiwUslc/PTxieAOKyd/AcdnDluGFuZL 2opSKM6pkP4MrClouQD2AoFgsmkCn5kP91slgWr1S0qE5HtESM7rWR5TF9ztU/47fy zH/h1upJeGpZOxPyaNm0cECPBknF62noMnsYIBD+uBEATPQWD+UdrM/cySgP5NHMRT FtShUmuCqvVbg== From: SeongJae Park To: cuiyangpei Cc: SeongJae Park , akpm@linux-foundation.org, damon@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, xiongping1@xiaomi.com Subject: Re: [PATCH 1/2] mm/damon/sysfs: Implement recording feature Date: Sun, 3 Dec 2023 19:37:45 +0000 Message-Id: <20231203193745.57035-1-sj@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231203054313.GA14316@cuiyangpei> References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: B099818001B X-Stat-Signature: tt6i6xomytz8y85ddpnhr39rwurftnri X-HE-Tag: 1701632270-643579 X-HE-Meta: U2FsdGVkX18hmyN30EmV3uCHqJIl8rL69T08dwLJ3NyuC0KIlakRuMn4gSLCB1w1ylPkbqB2VUch1iZhIQL9G5v81risU3AHLgT+E02w8EzWSkeIUiM4U34+msMlcvrcgOnXgE2+BOmO8NVDag5Xhgb404mSTVFgsH6TRkkBZdSJyc+AjXOKo9NjSS7ewZXeEa5bMpIwzCubfle/8tcluCRWVyAv+RD5KMj6SWPsaQOEjgQmoecjphqls7xVUHJ491946IOm69hkbKRYQ7V0uql8RUn/GRYagbwwoX2QkBzjtLm6oYZ0k9n8uaOTe7UvUBubVA1e0obXndhI0y/0VvcACtknJX7+HE6YeTPkMBdoabAMfjQTMcM3s/CrAGmtxbaH5nm3JtWrXtMKVR7+1Xs+XxIBlUc3wQjnDpNLaskmpMzuRit8FwaJPQuuGWrpmRakdEhRnpHZJ5IjP6R4H0gJ/6GVnYK4WO54x/ZLbsRG0/b/5S3bG6adhDHeThDzHhn2vGsJ4UpyeFWHTP6330W+eYTuBZy664kEHC1HO0WLhQ7qpaENPLP06W15gD1C6Ppc5YUMJWga8PdaXQN2cWPeTJeSf5cwrw6ozBcrfjDQYLJK+wYMFwbJizIUNnZsu23L4HxB/Dpd18OWpuI5IVBPIE3l/sBRWIMN3xJpcsoqAiCXgKIVjl4Hfk2LFPlAFN4TI+zXID/Nk93ty7qdCm8/mHn8fJg82q33lS2VE+fz6IMunutoHrFzTBQZOBoxA8Tos0tFvuj3fRU5hAyb7NKZH6o4PR1TYxmNUCXfkSAfNU4a8kWDfWcNO35DzNB0J0GPxf/+XKo0EmxkEE++mhfzSluvkDzzm1qivpQo0tVtdg9Ow1rZvLmSZkR+d23FWnoVEckdhRxYPG4dHjoitcdLFjpHbm/4NZibN/eb9YsNxOO4iwWPdKRVm+/m4qbIcIt6HlFW0S9E1XiCqIO 961FUsC5 wXZByRZuttdmp+w4OTPWCXsY8q+slUWEmO+rptw0KgasoaFsuxmkENtzdCgoMowUFDggE/v/J/X5+CIJZQUOsON3ZlGIeAS0iv9cNYgohrcn0eqeakBZ1zH5KUugcvu4omJqjWYXSJmvurQxW07UVLACNHBxlvxQBSt9HGBkv7g8vOgjQBK8WYzE/JdYZLGlNQ6XjxTUBBDLqkvMG+HT2B7leNGCskA875+RZSwz5V8OSpQ2yRlJiSADB8RBiO8jGao4BrH7wS6JYALYopG61LSDVr6n9OEcGXlU862I4zuPs/Ng4hRahA4WpXA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2023-12-03T13:43:13+08:00 cuiyangpei wrote: > On Fri, Dec 01, 2023 at 05:31:12PM +0000, SeongJae Park wrote: > > Hi Cuiyangpei, > > > > On Fri, 1 Dec 2023 20:25:07 +0800 cuiyangpei wrote: > > > > > On Thu, Nov 30, 2023 at 07:44:20PM +0000, SeongJae Park wrote: > > > > Hi Cuiyangpei, > > > > > > > > On Thu, 30 Nov 2023 17:14:26 +0800 cuiyangpei wrote: > > > > > > > > > Hi SeongJae, > > > > > > > > > > We also investigated the operation schemes you mentioned, but we don't > > > > > think it can fit our needs. > > > > > > > > > > On android, user will open many apps and switch between these apps as > > > > > needs. We hope to monitor apps' memory access only when they are on > > > > > foreground and record the memory access pattern when they are switched > > > > > to the background. > > > > > > > > > > When avaliable memory reaches a threshold, we will use these access > > > > > patterns with some strategies to recognize those memory that will have > > > > > little impact on user experience and to reclaim them proactively. > > > > > > > > > > I'm not sure I have clarified it clearly, if you still have questions > > > > > on this, please let us know. > > > > > > > > So, to my understanding, you expect applications may keep similar access > > > > pattern when they are in foreground, but have a different, less aggressive > > > > access pattern in background, and therefore reclaim memory based on the > > > > foreground-access pattern, right? > > > > > > > > > > Different apps may have different access pattern. On android, the apps will > > > join in freeze cgroup and be frozen after switch to the background. So we > > > monitor apps' memory access only when they are in foreground. > > > > Thank you for this enlightening me :) > > > > > > Very interesting idea, thank you for sharing! > > > > > > > > Then, yes, I agree current DAMOS might not that helpful for the situation, and > > > > this record feature could be useful for your case. > > > > > > > > That said, do you really need full recording of the monitoring results? If > > > > not, DAMOS provides DAMOS tried regions feature[1], which allows users get the > > > > monitoring results snapshot that include both frequency and recency of all > > > > regions in an efficient way. If single snapshot is not having enough > > > > information for you, you could collect multiple snapshots. > > > > > > > > You mentioned absence of Python on Android as a blocker of DAMOS use on the > > > > previous reply[2], but DAMOS tried regions feature is not depend on tracepoints > > > > or Python. > > > > > > > > Of course, I think you might already surveyed it but found some problems. > > > > Could you please share that in detail if so? > > > > > > > DAMOS tried regions feature you mentioned is not fully applicable. It needs to > > > apply schemes on regions. There is no available scheme we can use for our use > > > case. What we need is to return regions with access frequency and recency to > > > userspace for later use. > > > > > > Thank you for the answer, I understand your concern. One of the available > > DAMOS action is 'stat'[1], which does nothing but just count the statistic. > > Using DAMOS scheme for any access pattern with 'stat' action, you can extract > > the access pattern via DAMOS tried regions feature of DAMON sysfs interface, > > without making any unnecessary impact to the workload. > > > > Quote from [2]: > > > > The expected usage of this directory is investigations of schemes' behaviors, > > and query-like efficient data access monitoring results retrievals. For the > > latter use case, in particular, users can set the action as stat and set the > > access pattern as their interested pattern that they want to query. > > > > For example, you could > > > > # cd /sys/kernel/mm/damon/admin > > # > > # # populate directories > > # echo 1 > kdamonds/nr_kdamonds; echo 1 > kdamonds/0/contexts/nr_contexts; > > # echo 1 > kdamonds/0/contexts/0/schemes/nr_schemes > > # cd kdamonds/0/contexts/0/schemes/0 > > # > > # # set the access pattern for any case (max as 2**64 - 1), and action as stat > > # echo 0 > access_pattern/sz/min > > # echo 18446744073709551615 > access_pattern/sz/max > > # echo 0 > access_pattern/nr_accesses/min > > # echo 18446744073709551615 > access_pattern/nr_accesses/max > > # echo 0 > access_pattern/age/min > > # echo 18446744073709551615 > access_pattern/age/max > > # echo stat > action > > > > And this is how DAMON user-space tool is getting the snapshot with 'damo show' > > command[3]. > > > > Could this be used for your case? Please ask any question if you have :) > > > > [1] https://docs.kernel.org/admin-guide/mm/damon/usage.html#schemes-n > > [2] https://docs.kernel.org/admin-guide/mm/damon/usage.html#schemes-n-tried-regions, > > [3] https://github.com/awslabs/damo/blob/next/USAGE.md#damo-show > > Thank you for your detailed response, it is very helpful to us. We will look into it > and contact you if we have any questions. So glad to hear this. Please let me know if you have any questions or need any help :) Thanks, SJ > > > > > > > Thanks, > > SJ > > > > > > [1] https://docs.kernel.org/admin-guide/mm/damon/usage.html#schemes-n-tried-regions > > > > [2] https://lore.kernel.org/damon/20231129131315.GB12957@cuiyangpei/ > > > > > > > > > > > > Thanks, > > > > SJ > > > > > > > > > > > > > > Thanks.