From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D863C74A5B for ; Sun, 26 Mar 2023 07:21:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7FC1D900003; Sun, 26 Mar 2023 03:21:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7AC0F900002; Sun, 26 Mar 2023 03:21:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 699A1900003; Sun, 26 Mar 2023 03:21:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5E41F900002 for ; Sun, 26 Mar 2023 03:21:37 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1EF63140392 for ; Sun, 26 Mar 2023 07:21:37 +0000 (UTC) X-FDA: 80610204234.16.3D26EB2 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf20.hostedemail.com (Postfix) with ESMTP id 569521C0005 for ; Sun, 26 Mar 2023 07:21:35 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=qoEL8Z11; spf=pass (imf20.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679815295; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aewz/jXI+1WC5fJqifkc+E4NF6P6mRBmtyKsAvfmUzQ=; b=a/oG3LQFBeZpjKg5kZj5AisLrRyCQyRGEJ70X5MnBjw1wZIjAYnHUWyQoeY43QmnKYZaX5 Rpy7nK/iw1ES5xFXtAeIyMEQf9R+Y3Zu20aeQqmGB6qGh6fZ44DveFSuqLed7ahl893zbS rOLfL1uvYVYHHorbqBTvl3rTNiGpNxM= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=qoEL8Z11; spf=pass (imf20.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679815295; a=rsa-sha256; cv=none; b=AJBhOI7P3/AgndcdRrTfWCOrptYa2z+/KYAZMDVfAq9xcHOf/U+62I4Qs8wqVAxS7hXeai QY/fua6fzm4JzZ71k5sflKnk21v0hc8dIdR0YkFCwJcGMdCcXavr/Im90fqp4VpFFMul6F hoWbxltkwlBrWGtXUHXDgcx3PNAt0as= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3F78F60DE6; Sun, 26 Mar 2023 07:21:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 81E90C4339C; Sun, 26 Mar 2023 07:21:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679815293; bh=pxji88z27H0PgsSBucqTZmQTQyn/hGoPa6A+2B7kyW4=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=qoEL8Z115aro/jpVXBgeGYPwcL9BwfsejvL0g41x465R0UnGpA/zXlecUoU/yWn7y FLAWi9tECSA30YB/defwKA1auizlk4JfbZGWhFQpo1CW3Q+vuOHYMMPC0F/4h3Rt9G kIz/PUvmxiDxUuD7JkrkvEIrsyLpuDUAttNypHcnRQMO15p8DLcAmIKs4Nutqcbrvc jA8BlJz8+l4sSO0AOYNTkkAausD233l0C8SM0ckC1E61TrL9ZpOD/+wCpOcEf3so0c E2fA+c2E61WU8z6CZMzAhQ7uhKq01nPM+pessiFEIY4wTolXBimWtb+rTwW10a3shX DsFZ/xPvK0fUA== Date: Sun, 26 Mar 2023 10:21:19 +0300 From: Mike Rapoport To: Kyungsan Kim Cc: dan.j.williams@intel.com, lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-cxl@vger.kernel.org, a.manzanares@samsung.com, viacheslav.dubeyko@bytedance.com, ying.huang@intel.com Subject: Re: RE(2): FW: [LSF/MM/BPF TOPIC] SMDK inspired MM changes for CXL Message-ID: References: <641b7b2117d02_1b98bb294cb@dwillia2-xfh.jf.intel.com.notmuch> <20230323105105.145783-1-ks0204.kim@samsung.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230323105105.145783-1-ks0204.kim@samsung.com> X-Rspamd-Queue-Id: 569521C0005 X-Stat-Signature: 68ikyx1o5xsui6x8ez5iif99ctuzbyys X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1679815295-572527 X-HE-Meta: U2FsdGVkX19poYL5nzGAuTfbUYMPJMA/fnn2DrOp3tBgWhRxq0eyLoC7HIEWjuaniCWAhWLoFyg2GmsT83W2gCUaAXFZSYmSmWeRtu+SvX6FpNiXBTIv+Tvu6UemigPS/qyRF9ZA+WafADCeXlXImAurjK+EmkYf+08IFkhe0I2vZmGCJnyKmNshRkmsGOE02BzuOp9y6xVMGjCszTLu7XTXEMvjyL5YV25Uz6VyQMUVZblIonpnfVszepd4WNBhcE0Rnd2KCKdmV6cYoAVvOQURyvYGl9OVxKkaeJgBiSINDDVwESxAXy63y4884MVStfCed+PYtGxz0h5yyQ9WJ74RxUhGIxfHNfnaPvgdVXpr7/lqtDQgeJBLXzWscknYTaK6FxOnWvT7wJtTrzpQ3QfnziLtbjESlYoWny8l0RCB4GOTxR9hJScWvY14OSRpexEsLVkqFpB0MHHNM+VdqIgA0rVBo3Y/PYh2HaYpgxP8jylaHpQN44tneAp/IdZvv/XvN88ZGmy+9sHpMJFjgZtB+WCwCTD1XNGPndDKR0cmr8Xf5DvHw+BK10vk6/P5CC+DV1WOBgiH5Srics3Wx170YE6Y/2RqYuUuoy9cnwF0APhIXPzde+3InVLISzbWkpBZhnF5cQRYTwY5QZ27UVlHEH1MydzsXIvZki1itoh7aWliLzZEWO3VuKJyMdcFOHxZnVjP6ZBhnPZdmOZiAG/7w0Qm157JcGgbIsA27X00zlIRmqh0fszQmjHtkyQ8XRiFV2lX6Ib+kGCC8tobrwt38m6ytLrfp6a31vgIAPNt9A8S7904OJYQzoQECnzD01w8PO+sOuMRBtCr/i5e2zrs6eFxV2l/LrCRqdfWhUK2Hlk3MmCGrotWrV1bKW1qGQDujU5JP5T2fsEMssPFG3LSzGkudWSYv8byBOKTSPEghO+PUUCneCANnIt87t4IJRoQEZQGYrXDYxWMAAq prNTCdiC xMN3RuKjAqwSsrSF3141Y+B1b/iQFHdXYgeA72vrjFhPW9kYuQnvqg8pQUtgMJkwi119pm2c7S/EIBSqsGuftRw1Zis3L7oCVW9/bPaU6COKpJttTwhQQHDLQ8vcHsXv8vU0uuz00nXJYd1NrMntIE2cecpGBYckeibAQSjufcmy7BS4pMPB7cS1cZMAphL8oRU0CAqRCrlZ3Mgud3eEx6LiX0DpEJK97Ok9oe+TFVSYwGBMyHS63VP2OcI2RGf0WWFnvzcVmyulBJkbCJsCTZmN8dNf42rSCZHzgvXJosWPF3Mo7W6+iCtNCDw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi, On Thu, Mar 23, 2023 at 07:51:05PM +0900, Kyungsan Kim wrote: > I appreciate dan for the careful advice. > > >Kyungsan Kim wrote: > >[..] > >> >In addition to CXL memory, we may have other kind of memory in the > >> >system, for example, HBM (High Bandwidth Memory), memory in FPGA card, > >> >memory in GPU card, etc. I guess that we need to consider them > >> >together. Do we need to add one zone type for each kind of memory? > >> > >> We also don't think a new zone is needed for every single memory > >> device. Our viewpoint is the sole ZONE_NORMAL becomes not enough to > >> manage multiple volatile memory devices due to the increased device > >> types. Including CXL DRAM, we think the ZONE_EXMEM can be used to > >> represent extended volatile memories that have different HW > >> characteristics. > > > >Some advice for the LSF/MM discussion, the rationale will need to be > >more than "we think the ZONE_EXMEM can be used to represent extended > >volatile memories that have different HW characteristics". It needs to > >be along the lines of "yes, to date Linux has been able to describe DDR > >with NUMA effects, PMEM with high write overhead, and HBM with improved > >bandwidth not necessarily latency, all without adding a new ZONE, but a > >new ZONE is absolutely required now to enable use case FOO, or address > >unfixable NUMA problem BAR." Without FOO and BAR to discuss the code > >maintainability concern of "fewer degress of freedom in the ZONE > >dimension" starts to dominate. > > One problem we experienced was occured in the combination of hot-remove and kerelspace allocation usecases. > ZONE_NORMAL allows kernel context allocation, but it does not allow hot-remove because kernel resides all the time. > ZONE_MOVABLE allows hot-remove due to the page migration, but it only allows userspace allocation. > Alternatively, we allocated a kernel context out of ZONE_MOVABLE by adding GFP_MOVABLE flag. > In case, oops and system hang has occasionally occured because ZONE_MOVABLE can be swapped. > We resolved the issue using ZONE_EXMEM by allowing seletively choice of the two usecases. > As you well know, among heterogeneous DRAM devices, CXL DRAM is the first PCIe basis device, which allows hot-pluggability, different RAS, and extended connectivity. > So, we thought it could be a graceful approach adding a new zone and separately manage the new features. This still does not describe what are the use cases that require having kernel allocations on CXL.mem. I believe it's important to start with explanation *why* it is important to have kernel allocations on removable devices. > Kindly let me know any advice or comment on our thoughts. > > -- Sincerely yours, Mike.