From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_PATCH,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FC47C433DF for ; Fri, 21 Aug 2020 09:30:18 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4B236207DA for ; Fri, 21 Aug 2020 09:30:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="BRy2ijHc"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="EDCJ+xzB" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4B236207DA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=w8gC1QbhlJF0oPbScokzYtWG9S1twbKKkgYQ8MzMJPM=; b=BRy2ijHcGcozDTLuBm/iiFYH7 Hb/PJ6L+uN3DK1Rg3CAwGPIHNWWNt4cSWfmOWOR8CN35IaBztTJBtBLT/054VcBOjwm6JbwEttK1w WTYc0GwfXdnxNN5HhzdZZhfNg5A1wmaiGUbT2p8FUvE21FJOfzENiWESHvW/qWZqiuqKOkbhrV9FA HASqh2cSrQVcYTw5LXahTMRXbl6r3oE73qM8rGcv2UV/1xOOQO4gfylX/R7BuAGf4PIsDWD2yqEpo 93BSpa2Snu50o7oNspa1ynQLZAswikOTLbRDdPTTIg7h482W4SM0bAegibFRIDOyEnEMBoTkbMUkT rAUvxHD5Q==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k93KW-0000R1-8P; Fri, 21 Aug 2020 09:27:24 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k93KS-0000PS-PK for linux-arm-kernel@lists.infradead.org; Fri, 21 Aug 2020 09:27:22 +0000 Received: from willie-the-truck (236.31.169.217.in-addr.arpa [217.169.31.236]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 795BC207DA; Fri, 21 Aug 2020 09:27:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1598002040; bh=mzaSWGs1RnWzX6ePAs2v3rbn4wv6TFJSO1neT9Bhh/o=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=EDCJ+xzBJIbiQqq4qNPwD8SbSTRE5dS+0L2wiREdYf9hSx3ro2DPx+e7Tkvj1ereu 20F8ysfJ9kf2gCXozjvi2t0I8b+7rmca6fRFl2Tkt5kdSGiWcKaoWPXl6Lo2zUbUpR lkRouo+iBLwgQKgnjMKtxkyX7bU5S8RIhQo9bGLI= Date: Fri, 21 Aug 2020 10:27:14 +0100 From: Will Deacon To: "Song Bao Hua (Barry Song)" Subject: Re: [PATCH v6 1/2] dma-contiguous: provide the ability to reserve per-numa CMA Message-ID: <20200821092713.GD20255@willie-the-truck> References: <20200821022615.28596-1-song.bao.hua@hisilicon.com> <20200821022615.28596-2-song.bao.hua@hisilicon.com> <20200821084717.GA20255@willie-the-truck> <4ab78767553f48a584217063f6f24eb9@hisilicon.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <4ab78767553f48a584217063f6f24eb9@hisilicon.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200821_052720_984162_EED3D82E X-CRM114-Status: GOOD ( 39.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mike Rapoport , Steve Capper , "catalin.marinas@arm.com" , Jonathan Cameron , Linuxarm , "linux-kernel@vger.kernel.org" , "iommu@lists.linux-foundation.org" , Nicolas Saenz Julienne , "ganapatrao.kulkarni@cavium.com" , Andrew Morton , huangdaode , "robin.murphy@arm.com" , "hch@lst.de" , "linux-arm-kernel@lists.infradead.org" , "m.szyprowski@samsung.com" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, Aug 21, 2020 at 09:13:39AM +0000, Song Bao Hua (Barry Song) wrote: > > > > -----Original Message----- > > From: Will Deacon [mailto:will@kernel.org] > > Sent: Friday, August 21, 2020 8:47 PM > > To: Song Bao Hua (Barry Song) > > Cc: hch@lst.de; m.szyprowski@samsung.com; robin.murphy@arm.com; > > ganapatrao.kulkarni@cavium.com; catalin.marinas@arm.com; > > iommu@lists.linux-foundation.org; Linuxarm ; > > linux-arm-kernel@lists.infradead.org; linux-kernel@vger.kernel.org; > > huangdaode ; Jonathan Cameron > > ; Nicolas Saenz Julienne > > ; Steve Capper ; Andrew > > Morton ; Mike Rapoport > > Subject: Re: [PATCH v6 1/2] dma-contiguous: provide the ability to reserve > > per-numa CMA > > > > On Fri, Aug 21, 2020 at 02:26:14PM +1200, Barry Song wrote: > > > diff --git a/Documentation/admin-guide/kernel-parameters.txt > > b/Documentation/admin-guide/kernel-parameters.txt > > > index bdc1f33fd3d1..3f33b89aeab5 100644 > > > --- a/Documentation/admin-guide/kernel-parameters.txt > > > +++ b/Documentation/admin-guide/kernel-parameters.txt > > > @@ -599,6 +599,15 @@ > > > altogether. For more information, see > > > include/linux/dma-contiguous.h > > > > > > + pernuma_cma=nn[MG] > > > + [ARM64,KNL] > > > + Sets the size of kernel per-numa memory area for > > > + contiguous memory allocations. A value of 0 disables > > > + per-numa CMA altogether. DMA users on node nid will > > > + first try to allocate buffer from the pernuma area > > > + which is located in node nid, if the allocation fails, > > > + they will fallback to the global default memory area. > > > > What is the default behaviour if this option is not specified? Seems like > > that should be mentioned here. Just wanted to make sure you didn't miss this ^^ > > > > > diff --git a/kernel/dma/Kconfig b/kernel/dma/Kconfig > > > index 847a9d1fa634..db7a37ed35eb 100644 > > > --- a/kernel/dma/Kconfig > > > +++ b/kernel/dma/Kconfig > > > @@ -118,6 +118,16 @@ config DMA_CMA > > > If unsure, say "n". > > > > > > if DMA_CMA > > > + > > > +config DMA_PERNUMA_CMA > > > + bool "Enable separate DMA Contiguous Memory Area for each NUMA > > Node" > > > > I don't understand the need for this config option. If you have DMA_DMA and > > you have NUMA, why wouldn't you want this enabled? > > Christoph preferred this in previous patchset in order to be able to remove all of the code > in the text if users don't use pernuma CMA. Ok, I defer to Christoph here, but maybe a "default NUMA" might work? > > > + help > > > + Enable this option to get pernuma CMA areas so that devices like > > > + ARM64 SMMU can get local memory by DMA coherent APIs. > > > + > > > + You can set the size of pernuma CMA by specifying > > "pernuma_cma=size" > > > + on the kernel's command line. > > > + > > > comment "Default contiguous memory area size:" > > > > > > config CMA_SIZE_MBYTES > > > diff --git a/kernel/dma/contiguous.c b/kernel/dma/contiguous.c > > > index cff7e60968b9..89b95f10e56d 100644 > > > --- a/kernel/dma/contiguous.c > > > +++ b/kernel/dma/contiguous.c > > > @@ -69,6 +69,19 @@ static int __init early_cma(char *p) > > > } > > > early_param("cma", early_cma); > > > > > > +#ifdef CONFIG_DMA_PERNUMA_CMA > > > + > > > +static struct cma *dma_contiguous_pernuma_area[MAX_NUMNODES]; > > > +static phys_addr_t pernuma_size_bytes __initdata; > > > + > > > +static int __init early_pernuma_cma(char *p) > > > +{ > > > + pernuma_size_bytes = memparse(p, &p); > > > + return 0; > > > +} > > > +early_param("pernuma_cma", early_pernuma_cma); > > > +#endif > > > + > > > #ifdef CONFIG_CMA_SIZE_PERCENTAGE > > > > > > static phys_addr_t __init __maybe_unused > > cma_early_percent_memory(void) > > > @@ -96,6 +109,34 @@ static inline __maybe_unused phys_addr_t > > cma_early_percent_memory(void) > > > > > > #endif > > > > > > +#ifdef CONFIG_DMA_PERNUMA_CMA > > > +void __init dma_pernuma_cma_reserve(void) > > > +{ > > > + int nid; > > > + > > > + if (!pernuma_size_bytes) > > > + return; > > > > If this is useful (I assume it is), then I think we should have a non-zero > > default value, a bit like normal CMA does via CMA_SIZE_MBYTES. > > The patchet used to have a CONFIG_PERNUMA_CMA_SIZE in kernel/dma/Kconfig, > but Christoph was not comfortable with it: > https://lore.kernel.org/linux-iommu/20200728115231.GA793@lst.de/ > > Would you mind to hardcode the value in CONFIG_CMDLINE in arch/arm64/Kconfig as Christoph mentioned: > config CMDLINE > default "pernuma_cma=16M" > > If you also don't like the change in arch/arm64/Kconfig CMDLINE, I guess I > have to depend on users' setting in cmdline just like hugetlb_cma. Again, I defere to CHristophe for this code, so leave it like it is. However, the same argument applies to CMA_SIZE_MBYTES afaict, and I'm mainly looking for consistency. > > > + for_each_node_state(nid, N_ONLINE) { > > > > for_each_online_node() { > > > > > + int ret; > > > + char name[20]; > > > > 20? > > > > Ah, wait, this is copy-pasta from hugetlb_cma_reserve(). Can you factor out > > the common parts at all? > > Actually I have a "#define CMA_MAX_NAME 64" in this commit: > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=18e98e56f440 > > the 20 in hugetlb_cma_reserve() was also made by me. If you are not comfortable, I can > move to CMA_MAX_NAME. do you think it does really matter here? 20 seems to be long > enough for this scenario. Using CMA_MAX_NAME seems sensible to me, although I'm still a bit wary about the code duplication between this and the hugetlb code. Will _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel