From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B87DC433E4 for ; Tue, 28 Jul 2020 11:54:05 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1AB68206D7 for ; Tue, 28 Jul 2020 11:54:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="aJ/NFP5B" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1AB68206D7 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=lst.de Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Sk6sLFzn9zlwuK+tsm8DZCBVNgH/K4S3vmoP7ZCHx6s=; b=aJ/NFP5Btw+x8CFxMqYIe+x5B bU2m6gDALPn99KlcIBu04fs2n+K/80t78J3lQANk0phRyF1h1NHIcZPanJmR0/6+oByz70dVP1KyI dyN7P4zaNVq1Qdcy0SxAOKXT3sck4pud1Iehzwongjuhe22cF0mANyT42HCxsdhjV2DjSN+QcUWuw iwnpwV1BWlSgy3BHSUD62crMqYVPxYMH4GfEU+YVpemuDyW0Pa/txQ+LGWIZKUEdxFeMLO7BPKYQY OAXr6uvXulOgMNHa4fLtx3DoVZRwHLSjwwqxuyMnR92NOl2/NaXuMMjmqZ0G9+JY9uvlB9h6FQgMS qbLYgCzwA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k0O9u-0007Tk-D7; Tue, 28 Jul 2020 11:52:38 +0000 Received: from verein.lst.de ([213.95.11.211]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k0O9r-0007T1-Oh for linux-arm-kernel@lists.infradead.org; Tue, 28 Jul 2020 11:52:36 +0000 Received: by verein.lst.de (Postfix, from userid 2407) id 0344268B05; Tue, 28 Jul 2020 13:52:32 +0200 (CEST) Date: Tue, 28 Jul 2020 13:52:31 +0200 From: Christoph Hellwig To: Barry Song Subject: Re: [PATCH v4 1/2] dma-direct: provide the ability to reserve per-numa CMA Message-ID: <20200728115231.GA793@lst.de> References: <20200723131344.41472-1-song.bao.hua@hisilicon.com> <20200723131344.41472-2-song.bao.hua@hisilicon.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20200723131344.41472-2-song.bao.hua@hisilicon.com> User-Agent: Mutt/1.5.17 (2007-11-01) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200728_075236_050149_38AA98F4 X-CRM114-Status: GOOD ( 26.93 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: catalin.marinas@arm.com, Mike Rapoport , Steve Capper , robin.murphy@arm.com, Jonathan Cameron , linuxarm@huawei.com, linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, Nicolas Saenz Julienne , prime.zeng@hisilicon.com, ganapatrao.kulkarni@cavium.com, Andrew Morton , huangdaode@huawei.com, will@kernel.org, hch@lst.de, linux-arm-kernel@lists.infradead.org, m.szyprowski@samsung.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, Jul 24, 2020 at 01:13:43AM +1200, Barry Song wrote: > +config CMA_PERNUMA_SIZE_MBYTES > + int "Size in Mega Bytes for per-numa CMA areas" > + depends on NUMA > + default 16 if ARM64 > + default 0 > + help > + Defines the size (in MiB) of the per-numa memory area for Contiguous > + Memory Allocator. Every numa node will get a separate CMA with this > + size. If the size of 0 is selected, per-numa CMA is disabled. I'm still not a fan of the config option. You can just hardcode the value in CONFIG_CMDLINE based on the kernel parameter. Also I wonder if a way to expose this in the device tree might be useful, but people more familiar with the device tree and the arm code will have to chime in on that. > struct cma *dma_contiguous_default_area; > +static struct cma *dma_contiguous_pernuma_area[MAX_NUMNODES]; > > /* > * Default global CMA area size can be defined in kernel's .config. > @@ -44,6 +51,8 @@ struct cma *dma_contiguous_default_area; > */ > static const phys_addr_t size_bytes __initconst = > (phys_addr_t)CMA_SIZE_MBYTES * SZ_1M; > +static phys_addr_t pernuma_size_bytes __initdata = > + (phys_addr_t)CMA_SIZE_PERNUMA_MBYTES * SZ_1M; > static phys_addr_t size_cmdline __initdata = -1; > static phys_addr_t base_cmdline __initdata; > static phys_addr_t limit_cmdline __initdata; > @@ -69,6 +78,13 @@ static int __init early_cma(char *p) > } > early_param("cma", early_cma); > > +static int __init early_pernuma_cma(char *p) > +{ > + pernuma_size_bytes = memparse(p, &p); > + return 0; > +} > +early_param("pernuma_cma", early_pernuma_cma); > + > #ifdef CONFIG_CMA_SIZE_PERCENTAGE > > static phys_addr_t __init __maybe_unused cma_early_percent_memory(void) > @@ -96,6 +112,33 @@ static inline __maybe_unused phys_addr_t cma_early_percent_memory(void) > > #endif > > +void __init dma_pernuma_cma_reserve(void) > +{ > + int nid; > + > + if (!pernuma_size_bytes) > + return; > + > + for_each_node_state(nid, N_MEMORY) { > + int ret; > + char name[20]; > + > + snprintf(name, sizeof(name), "pernuma%d", nid); > + ret = cma_declare_contiguous_nid(0, pernuma_size_bytes, 0, 0, > + 0, false, name, > + &dma_contiguous_pernuma_area[nid], > + nid); This adds a > 80 char line. > struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp) > { > + int nid = dev_to_node(dev); > + > /* CMA can be used only in the context which permits sleeping */ > if (!gfpflags_allow_blocking(gfp)) > return NULL; > if (dev->cma_area) > return cma_alloc_aligned(dev->cma_area, size, gfp); > - if (size <= PAGE_SIZE || !dma_contiguous_default_area) > + if (size <= PAGE_SIZE) > return NULL; > + > + if ((nid != NUMA_NO_NODE) && !(gfp & (GFP_DMA | GFP_DMA32))) { No need for the braces around the nid check. > + struct cma *cma = dma_contiguous_pernuma_area[nid]; > + struct page *page; > + > + if (cma) { > + page = cma_alloc_aligned(cma, size, gfp); > + if (page) > + return page; > + } > + } > + > return cma_alloc_aligned(dma_contiguous_default_area, size, gfp); This seems to have lost the dma_contiguous_default_area NULL check. > + /* if dev has its own cma, free page from there */ > + if (dev->cma_area) { > + if (cma_release(dev->cma_area, page, PAGE_ALIGN(size) >> PAGE_SHIFT)) > + return; Another overly long line. > + } else { > + /* > + * otherwise, page is from either per-numa cma or default cma > + */ > + if (cma_release(dma_contiguous_pernuma_area[page_to_nid(page)], > + page, PAGE_ALIGN(size) >> PAGE_SHIFT)) > + return; > + > + if (cma_release(dma_contiguous_default_area, page, > + PAGE_ALIGN(size) >> PAGE_SHIFT)) > + return; > + } I'd introduce a count variable for the value of "PAGE_ALIGN(size) >> PAGE_SHIFT" to clean al lthis up a bit. Also please add a CONFIG_PERCPU_DMA_CMA config variable so that we don't build this code for the vast majority of users that don't need it. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel