From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F24FC76188 for ; Wed, 5 Apr 2023 18:13:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 97CBC6B0071; Wed, 5 Apr 2023 14:13:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 92CC76B0074; Wed, 5 Apr 2023 14:13:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 81BEB6B0075; Wed, 5 Apr 2023 14:13:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 741986B0071 for ; Wed, 5 Apr 2023 14:13:00 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 4512E1A0857 for ; Wed, 5 Apr 2023 18:13:00 +0000 (UTC) X-FDA: 80648133720.21.331EFFB Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf08.hostedemail.com (Postfix) with ESMTP id A27F7160005 for ; Wed, 5 Apr 2023 18:12:58 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=jMd1255a; spf=none (imf08.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680718378; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=A1OvYUgP4FG9EQlktdUAjWwsZKNpPiqegresn/FLTXw=; b=hmdM0wehd2JSY7GiEonYBgMW6x+U6/9F2KQoQ9kzBiRERE7mepvabybaPIa2C2l7J25DAy Y5e9PEUz2UH54mn31A/GAKg6uqwIAg5maWws5JRMX2rHOIPhtesHB57l0qXr6aGtNeKDs6 AWRg/Xm8g3ZTI5iG4O4IXIsS/aIQzvs= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=jMd1255a; spf=none (imf08.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680718378; a=rsa-sha256; cv=none; b=hLk81QnXyd5SybIjnC2p/go0OrbBPlplAzWfjaj9RXRqxURrqMHKxAck7/vvDA1Oh8CeqL bIkwecJ+5dnPkVtwg3kSeEGX2UW8tRgzWqBpSstOE7j9Jn5M1w5Us/rcYBr6ou7Asseajl nRCRhP7t9j7oXKT61TqiY+YdYgYE0z8= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=A1OvYUgP4FG9EQlktdUAjWwsZKNpPiqegresn/FLTXw=; b=jMd1255as8B3cu19099m7uTzR4 +PK3TbWYZOHCxqYFSPrIXmyCD+B9rjxZjSVmYctrshO+gGCqvA3Ad5sT0Oy5UnH+sVK0a98/q66T3 eILt8SACBMmEEglX6LWK3NejoAFlReV92ipO+TWoa61avFcm3OTQ55cf1NBn8sae+VRYgoiRCwMNk XFKkinkcjWg4OFuW9KBud6Y41ITwsfRPduZ6claAyA2sfc2Zf9gmUupYKiNuD25MMNNmGt93rnUex COVlRUnCECa4eNvW4vxZzsq+nvl+LdZoyxERjiTdbwHd+Z+yf6Fb3qOyra1SPy7r0psMH9blQHOtE 7kgDwY1w==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pk7ck-00Gcqk-Uw; Wed, 05 Apr 2023 18:12:46 +0000 Date: Wed, 5 Apr 2023 19:12:46 +0100 From: Matthew Wilcox To: Dan Williams Cc: Kyungsan Kim , lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-cxl@vger.kernel.org, a.manzanares@samsung.com, viacheslav.dubeyko@bytedance.com, seungjun.ha@samsung.com, wj28.lee@samsung.com Subject: Re: Re: Re: RE(2): FW: [LSF/MM/BPF TOPIC] SMDK inspired MM changes for CXL Message-ID: References: <20230405020027.413578-1-ks0204.kim@samsung.com> <642cfda9ccd64_21a8294fd@dwillia2-xfh.jf.intel.com.notmuch> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <642cfda9ccd64_21a8294fd@dwillia2-xfh.jf.intel.com.notmuch> X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: q9tx47ggwb3g613koauxbbdopumw4p6g X-Rspamd-Queue-Id: A27F7160005 X-HE-Tag: 1680718378-792082 X-HE-Meta: U2FsdGVkX18ctMJwc2eYMNHHP+RL4Pn3TQP3p727ljZTX8dMu0TS8X3D04FpQUj58XzZ0Mc0VrJUibVZQD6Xdo+S6JCzmUTQOYrlVJUCWnbj98rw5iyPC0hhAk0vsZp4vRd+KrEjpy9g4DzYurwmi1F2wjdiOeXBeOtrZ/9QGHEecRDkRtOOucyi1z4l9k52ADI5nDdAoFGtBuAvG3nCnVK7PIH5fKN/YdABkzdhLpFfMmv3EKUNJNu2ZNgRkCeXkuuUR/DQxnj726blAGGCWQhNvFFWcioaYKNcqHfvy3X1CJlQ6FORbi7P7am80QieXacDSa5sptNKzwW28luCW4fOCd3GZ57JQRapfklYslRl7M3L3bQwq0Lcr3dCzLMgQ+GwluhcyBqwk8Orz19tLlSaeMLvyjnhA+f6COnoph/v9DJlOAODwEmTdJojSRC/LaEeXiAlJroaTm/lO4T2Q8IsYSqkwXOTVsD7EKmO0PvD8ARkXTRb3n9foG1kdM7U/9E4fC0HZ6W7CIoXJhsDDxMEGwyEYn40mGUR04lyG116AYF819wD07U+TolaPsk1m89A27Alo9Hkp/DUWMMPLimeFio4V4c75lA+L5qbNqzITUj0+/N2xm/7+WDkILtpzCS4p4d3KXfmH5NXMTN1uirj55h7QawJiLxLhRY5yL8TPo2Ql/FIYqCEGtCI5P1YGS8nTPSX7kECaInHsF3D/AJM0BlaLTKtT5LfSv6ory+wYEnhydJS4qyrq7din0VNQlRB6wyAR9Mn8eaNv7FJMA/+Uo8C73qGMFU40FjKeMPaRRg3ierV3WS+q5yXfHdGJndhu+W/D+6h3eQ3N8IeNvzYnHmoesYhzgRtfp6BPCkp5rZvcvUpILBnCAx0hBF3SfupoCeHv3u9YNHER//MBoxc8Ia1cn8Ci6Iy3XSLVLDDnrPmfOhUyOdFFeI8EmYSK0CvCymQcLtgJ7FKQgg D5ufMLsL mZsqyMNmt4b07q37bd2DnElBX6i3Q5xApiyAs+55lC1ISRNFCIIdjCCyrCac3FGE/Hplgx5yUFbciglAWdXANM1rdDvivMwU65wGhQ6lfEgktKSJVxbj6JY6hDzO6nTxzNQtXkFDyBch1M4L4RVpUUNocN7dDEWWOsKzOg3v3g2TNAt5az+areauUPxBffcQwyjhXxNYmJ0VNGkC6TIvYGsTb0JdOYS0/egmT X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Apr 04, 2023 at 09:48:41PM -0700, Dan Williams wrote: > Kyungsan Kim wrote: > > We know the situation. When a CXL DRAM channel is located under ZONE_NORMAL, > > a random allocation of a kernel object by calling kmalloc() siblings makes the entire CXL DRAM unremovable. > > Also, not all kernel objects can be allocated from ZONE_MOVABLE. > > > > ZONE_EXMEM does not confine a movability attribute(movable or unmovable), rather it allows a calling context can decide it. > > In that aspect, it is the same with ZONE_NORMAL but ZONE_EXMEM works for extended memory device. > > It does not mean ZONE_EXMEM support both movability and kernel object allocation at the same time. > > In case multiple CXL DRAM channels are connected, we think a memory consumer possibly dedicate a channel for movable or unmovable purpose. > > > > I want to clarify that I expect the number of people doing physical CXL > hotplug of whole devices to be small compared to dynamic capacity > devices (DCD). DCD is a new feature of the CXL 3.0 specification where a > device maps 1 or more thinly provisioned memory regions that have > individual extents get populated and depopulated by a fabric manager. > > In that scenario there is a semantic where the fabric manager hands out > 100G to a host and asks for it back, it is within the protocol that the > host can say "I can give 97GB back now, come back and ask again if you > need that last 3GB". Presumably it can't give back arbitrary chunks of that 100GB? There's some granularity that's preferred; maybe on 1GB boundaries or something? > In other words even pinned pages in ZONE_MOVABLE are not fatal to the > flow. Alternatively, if a deployment needs 100% guarantees that the host > will return all the memory it was assigned when asked there is always > the option to keep that memory out of the page allocator and just access > it via a device. That's the role device-dax plays for "dedicated" memory > that needs to be set aside from kernel allocations. > > This is to say something like ZONE_PREFER_MOVABLE semantics can be > handled within the DCD protocol, where 100% unpluggability is not > necessary and 97% is good enough. This certainly makes life better (and rather more like hypervisor shrinking than like DIMM hotplug), but I think fragmentation may well result in "only 3GB of 100GB allocated" will result in being able to return less than 50% of the memory, depending on granule size and exactly how the allocations got chunked.