From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751504AbdALL5Y (ORCPT ); Thu, 12 Jan 2017 06:57:24 -0500 Received: from mout.kundenserver.de ([212.227.126.130]:55600 "EHLO mout.kundenserver.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751021AbdALL5W (ORCPT ); Thu, 12 Jan 2017 06:57:22 -0500 From: Arnd Bergmann To: Sagi Grimberg Cc: Christoph Hellwig , Nikita Yushchenko , linux-arm-kernel@lists.infradead.org, Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, linux-renesas-soc@vger.kernel.org, Simon Horman , linux-pci@vger.kernel.org, Bjorn Helgaas , artemi.ivanov@cogentembedded.com, Keith Busch , Jens Axboe , linux-nvme@lists.infradead.org Subject: Re: NVMe vs DMA addressing limitations Date: Thu, 12 Jan 2017 12:56:07 +0100 Message-ID: <3306663.hKmLLq1hhl@wuerfel> User-Agent: KMail/5.1.3 (Linux/4.4.0-34-generic; KDE/5.18.0; x86_64; ; ) In-Reply-To: <80676b35-121b-0462-23fc-ed5608e1e671@grimberg.me> References: <1483044304-2085-1-git-send-email-nikita.yoush@cogentembedded.com> <20170110144839.GB27156@lst.de> <80676b35-121b-0462-23fc-ed5608e1e671@grimberg.me> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Provags-ID: V03:K0:W6Q35mG9bmXOabYyfEkl2frpdXyz7EWZXF6kQ1WsHobQT/xPv7u LClr6GBcI9KIWwkzSu3V7WMbVJmRKer0r6+AXoHkYxNEsriw4Z/ECzFh3M9yzijhrQQrPbh vsEtSIguZ8TINqWPS3EAxkF1kxvd3ZSBmUIdgMbCgQAugLhjNfmzdPwWno8gK5lGd7hylDR 0YdgleCoHLGL8LsZpxVWA== X-UI-Out-Filterresults: notjunk:1;V01:K0:giObhMfPCaY=:hdrV40ejzkU58bR0TwPq2h GLSHDYNZgGGVh70ux4cdeZZ1Pz7D2o11OdREUd3sXQl/sAoBTI+oe3WucvKRVYYYhVHzAopza F7BAwrJFD+eXuybw4pcNmJ+4LdEzmkNQYAWnsCGh7coSPrmslOSGJSXK6JpuTtioCvkSk2p0k xawgC8cynipqo+pj8vqG52eJ0XhfiLA59UfOOHOxyAKH2zCNm8S8ZoxB565jbBXPtoaAK1JW/ Fy9j+2/xlfyriDjVpmB3Q28KD9M58ayAshgRt1dq2wKBW8201CcCC1C1dpp6uTeepo/NSDktv cuY5jrJpEOJnRv/YtP+0PV69uhKtMYXAh84JbSaBGfldUHkvFEdQ1So32Kt5o2W8JepJfNDve tDkHVa3RIUjBEs+gNonlSSNRqpc588HAQ7g66bvMFFXHxMfJqKg8wWwCRZqS5yNU7hCyximqX f9nC1cjmLDJqK2XQMu0vnqH/yOUP9waQq/+7cYIahts7/NaKUHg5k2YCdIksUheeoJrfRU+6N /cEOVQnsJPv7lF5nvyz6SGIfGn2JWEdaefRuB5fEYpeIcia7zX721ywHmFbzajObfScEq5DMw mvW04+4kRJk0+ljooAjbdedQSDjqsQitsrCKdsXwOgHOSA3NbzzW+LsogfrmJ7TzHa+4j1OM7 nA/vSnJuYgX4YECnAqvNFmT+Qrvo8RZhAT8ySr1I8jGME6Ge6Ck/T+tT32hfRSlO6kZaKb3J5 xDmHOiFCkX5siSLd Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thursday, January 12, 2017 12:09:11 PM CET Sagi Grimberg wrote: > >> Another workaround me might need is to limit amount of concurrent DMA > >> in the NVMe driver based on some platform quirk. The way that NVMe works, > >> it can have very large amounts of data that is concurrently mapped into > >> the device. > > > > That's not really just NVMe - other storage and network controllers also > > can DMA map giant amounts of memory. There are a couple aspects to it: > > > > - dma coherent memoery - right now NVMe doesn't use too much of it, > > but upcoming low-end NVMe controllers will soon start to require > > fairl large amounts of it for the host memory buffer feature that > > allows for DRAM-less controller designs. As an interesting quirk > > that is memory only used by the PCIe devices, and never accessed > > by the Linux host at all. > > Would it make sense to convert the nvme driver to use normal allocations > and use the DMA streaming APIs (dma_sync_single_for_[cpu|device]) for > both queues and future HMB? That is an interesting question: We actually have the "DMA_ATTR_NO_KERNEL_MAPPING" for this case, and ARM implements it in the coherent interface, so that might be a good fit. Implementing it in the streaming API makes no sense since we already have a kernel mapping here, but using a normal allocation (possibly with DMA_ATTR_NON_CONSISTENT or DMA_ATTR_SKIP_CPU_SYNC, need to check) might help on other architectures that have limited amounts of coherent memory and no CMA. Another benefit of the coherent API for this kind of buffer is that we can use CMA where available to get a large consecutive chunk of RAM on architectures without an IOMMU when normal memory is no longer available because of fragmentation. Arnd