From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3FC4C352AA for ; Wed, 2 Oct 2019 18:28:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 912B3218DE for ; Wed, 2 Oct 2019 18:28:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nr6ZKcXN" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728787AbfJBS2L (ORCPT ); Wed, 2 Oct 2019 14:28:11 -0400 Received: from mail-pg1-f196.google.com ([209.85.215.196]:43682 "EHLO mail-pg1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726076AbfJBS2L (ORCPT ); Wed, 2 Oct 2019 14:28:11 -0400 Received: by mail-pg1-f196.google.com with SMTP id v27so12326847pgk.10; Wed, 02 Oct 2019 11:28:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=izcdKMTlTPXdg1rZpK9PkNPTeAL5ACc5p0aT6ECn+3o=; b=nr6ZKcXN/729CI5YvKkqVc0Bz2wyp7AqMiNJbnneIbJ2S8a6/RpgT+vQxveooa1g8o Tzstjrebdj0ch6CBEQIhLHedt+Meqh8poQODSN2lg0OykjXXZCvv3ItaPXFO7ZImXqcW RuKmutNc32nfhTfdTSE+Gn+FP0hyvVehIb12ZNHr5rdWDezRIrhU/4qCY2koq02vv9hS 0JYsKwzPEkKLeHC0o1yMLr4lNeC7nIidqjJQlgZ17X3hwG7uMGUGlb0FYDKstybAoxSN PuzGKEISFL71Mt9LOzHTLH19Bi/ikZqtSvsTTAUBs/9/pSrhf3mQrJw/Bi+Y2mWPIur/ yzHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=izcdKMTlTPXdg1rZpK9PkNPTeAL5ACc5p0aT6ECn+3o=; b=pmsHGoI1ePqZE8TLIu0Dey8pVqh1NIIj/A8bMidf6zGs1MH2FE1AqnpVY91B+3CLt4 8BheWaKNwiIe64AuVFIJOqTmRF+wchCvko/tc6ceKJuIY8xNxd18XG4w7zxqtj7bOmQM 23z+BEWQzE3U6Kc5/wR3Ro7gBJr/RNPvTtzVCBwWjkobkxurHdBt8LgZEXKKObj/ijfp V1Z9ByZuI2uY3INWNDsKubaNPWcLw4eDL/a8pIDp789JIRi43c9zAMNUZiMvu13fc4ug oD3vwMBWsm5OejBjBXsSYU+mE8l75Yv7FuI+nl6TSMi5293fIXUyqTuPChzLR1LiXXB7 3y7Q== X-Gm-Message-State: APjAAAWttjPq1lnPKLQ3CMBX0r/izgeFIFB8DCBRwvf1KTLBlY3nuQVw rLq29wRs+YQEyzQLoW9cqqNMYGeN X-Google-Smtp-Source: APXvYqwrs3FB89RH0+kgXBF3ejwmYPZLTgHaLNuUiXE1AjcShSZR76+3xPYW28RCBcE0p63294782w== X-Received: by 2002:a17:90a:c214:: with SMTP id e20mr5758828pjt.81.1570040889544; Wed, 02 Oct 2019 11:28:09 -0700 (PDT) Received: from [10.69.78.41] ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id b14sm162486pfi.95.2019.10.02.11.28.07 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 02 Oct 2019 11:28:08 -0700 (PDT) Subject: Re: [PATCH 00/11] of: Fix DMA configuration for non-DT masters To: Robin Murphy , Nicolas Saenz Julienne , Rob Herring Cc: "moderated list:ARM/FREESCALE IMX / MXC ARM ARCHITECTURE" , devicetree@vger.kernel.org, Matthias Brugger , Frank Rowand , linux-arm-msm , linux-wireless , "linux-kernel@vger.kernel.org" , dri-devel , etnaviv@lists.freedesktop.org, "open list:DMA GENERIC OFFLOAD ENGINE SUBSYSTEM" , Stefan Wahren , james.quinlan@broadcom.com, linux-pci@vger.kernel.org, linux-tegra@vger.kernel.org, xen-devel@lists.xenproject.org, Dan Williams , freedreno , Linux Media Mailing List References: <20190924181244.7159-1-nsaenzjulienne@suse.de> <43fb5fe1de317d65a4edf592f88ea150c6e3b8cc.camel@suse.de> <307b988d0c67fb1c42166eca12742bcfda09d92d.camel@suse.de> From: Florian Fainelli Message-ID: Date: Wed, 2 Oct 2019 11:28:06 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.1.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit Sender: dmaengine-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org On 9/26/2019 4:20 AM, Robin Murphy wrote: > On 2019-09-26 11:44 am, Nicolas Saenz Julienne wrote: >>>>>> Robin, have you looked into supporting multiple dma-ranges? It's the >>>>>> next thing >>>>>> we need for BCM STB's PCIe. I'll have a go at it myself if nothing >>>>>> is in >>>>>> the >>>>>> works already. >>>>> >>>>> Multiple dma-ranges as far as configuring inbound windows should work >>>>> already other than the bug when there's any parent translation. But if >>>>> you mean supporting multiple DMA offsets and masks per device in the >>>>> DMA API, there's nothing in the works yet. >> >> Sorry, I meant supporting multiple DMA offsets[1]. I think I could >> still make >> it with a single DMA mask though. > > The main problem for supporting that case in general is the disgusting > carving up of the physical memory map you may have to do to guarantee > that a single buffer allocation cannot ever span two windows with > different offsets. I don't think we ever reached a conclusion on whether > that was even achievable in practice. It is with the Broadcom STB SoCs which have between 1 and 3 memory controllers depending on the SoC, and multiple dma-ranges cells for PCIe as a consequence. Each memory controller has a different physical address aperture in the CPU's physical address map (e.g.: MEMC0 is 0x0 - 0x3fff_ffff, MEMC1 0x4000_0000 - 0x7ffff_ffff and MEMC2 0x8000_0000 - 0xbfff_ffff, not counting the extension regions above 4GB), and while the CPU is scheduled and arbitrated the same way across all memory controllers (thus making it virtually UMA, almost) having a buffer span two memory controllers would be problematic because the memory controllers do not know how to guarantee the transaction ordering and buffer data consistency in both DRAM itself and for other memory controller clients, like PCIe. We historically had to reserve the last 4KB of each memory controller to avoid problematic controllers like EHCI to prefetch beyond the end of a memory controller's populated memory and that also incidentally takes care of never having a buffer cross a controller boundary. Either you can allocate the entire buffer on a given memory controller, or you cannot allocate memory at all on that zone/region and another one must be found (or there is no more memory and there is a genuine OOM). The way we reserve memory right now is based on the first patch submitted by Jim: https://lore.kernel.org/patchwork/patch/988469/ whereby we read the memory node's "reg" property and we map the physical addresses to the memory controller configuration read from the specific registers in the CPU's Bus Interface Unit (where the memory controller apertures are architecturally defined) and then we use that to call memblock_reserve() (not part of that patch, it should be though). -- Florian From mboxrd@z Thu Jan 1 00:00:00 1970 From: Florian Fainelli Subject: Re: [PATCH 00/11] of: Fix DMA configuration for non-DT masters Date: Wed, 2 Oct 2019 11:28:06 -0700 Message-ID: References: <20190924181244.7159-1-nsaenzjulienne@suse.de> <43fb5fe1de317d65a4edf592f88ea150c6e3b8cc.camel@suse.de> <307b988d0c67fb1c42166eca12742bcfda09d92d.camel@suse.de> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Return-path: In-Reply-To: Content-Language: en-US Sender: linux-kernel-owner@vger.kernel.org To: Robin Murphy , Nicolas Saenz Julienne , Rob Herring Cc: "moderated list:ARM/FREESCALE IMX / MXC ARM ARCHITECTURE" , devicetree@vger.kernel.org, Matthias Brugger , Frank Rowand , linux-arm-msm , linux-wireless , "linux-kernel@vger.kernel.org" , dri-devel , etnaviv@lists.freedesktop.org, "open list:DMA GENERIC OFFLOAD ENGINE SUBSYSTEM" , Stefan Wahren , james.quinlan@broadcom.com, linux-pci@vger.kernel.org, linux-tegra@vger.kernel.org, xen-devel@lists.xenproject.org, Dan Williams , freedreno , Linux List-Id: linux-tegra@vger.kernel.org On 9/26/2019 4:20 AM, Robin Murphy wrote: > On 2019-09-26 11:44 am, Nicolas Saenz Julienne wrote: >>>>>> Robin, have you looked into supporting multiple dma-ranges? It's the >>>>>> next thing >>>>>> we need for BCM STB's PCIe. I'll have a go at it myself if nothing >>>>>> is in >>>>>> the >>>>>> works already. >>>>> >>>>> Multiple dma-ranges as far as configuring inbound windows should work >>>>> already other than the bug when there's any parent translation. But if >>>>> you mean supporting multiple DMA offsets and masks per device in the >>>>> DMA API, there's nothing in the works yet. >> >> Sorry, I meant supporting multiple DMA offsets[1]. I think I could >> still make >> it with a single DMA mask though. > > The main problem for supporting that case in general is the disgusting > carving up of the physical memory map you may have to do to guarantee > that a single buffer allocation cannot ever span two windows with > different offsets. I don't think we ever reached a conclusion on whether > that was even achievable in practice. It is with the Broadcom STB SoCs which have between 1 and 3 memory controllers depending on the SoC, and multiple dma-ranges cells for PCIe as a consequence. Each memory controller has a different physical address aperture in the CPU's physical address map (e.g.: MEMC0 is 0x0 - 0x3fff_ffff, MEMC1 0x4000_0000 - 0x7ffff_ffff and MEMC2 0x8000_0000 - 0xbfff_ffff, not counting the extension regions above 4GB), and while the CPU is scheduled and arbitrated the same way across all memory controllers (thus making it virtually UMA, almost) having a buffer span two memory controllers would be problematic because the memory controllers do not know how to guarantee the transaction ordering and buffer data consistency in both DRAM itself and for other memory controller clients, like PCIe. We historically had to reserve the last 4KB of each memory controller to avoid problematic controllers like EHCI to prefetch beyond the end of a memory controller's populated memory and that also incidentally takes care of never having a buffer cross a controller boundary. Either you can allocate the entire buffer on a given memory controller, or you cannot allocate memory at all on that zone/region and another one must be found (or there is no more memory and there is a genuine OOM). The way we reserve memory right now is based on the first patch submitted by Jim: https://lore.kernel.org/patchwork/patch/988469/ whereby we read the memory node's "reg" property and we map the physical addresses to the memory controller configuration read from the specific registers in the CPU's Bus Interface Unit (where the memory controller apertures are architecturally defined) and then we use that to call memblock_reserve() (not part of that patch, it should be though). -- Florian From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.0 required=3.0 tests=DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 128D3C10F14 for ; Wed, 2 Oct 2019 18:28:31 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DA740218DE for ; Wed, 2 Oct 2019 18:28:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nr6ZKcXN" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DA740218DE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iFjMD-0007dh-Iv; Wed, 02 Oct 2019 18:28:13 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iFjMB-0007dY-CA for xen-devel@lists.xenproject.org; Wed, 02 Oct 2019 18:28:11 +0000 X-Inumbo-ID: 62ecdedc-e542-11e9-97fb-bc764e2007e4 Received: from mail-pf1-x444.google.com (unknown [2607:f8b0:4864:20::444]) by localhost (Halon) with ESMTPS id 62ecdedc-e542-11e9-97fb-bc764e2007e4; Wed, 02 Oct 2019 18:28:10 +0000 (UTC) Received: by mail-pf1-x444.google.com with SMTP id y5so10916795pfo.4 for ; Wed, 02 Oct 2019 11:28:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=izcdKMTlTPXdg1rZpK9PkNPTeAL5ACc5p0aT6ECn+3o=; b=nr6ZKcXN/729CI5YvKkqVc0Bz2wyp7AqMiNJbnneIbJ2S8a6/RpgT+vQxveooa1g8o Tzstjrebdj0ch6CBEQIhLHedt+Meqh8poQODSN2lg0OykjXXZCvv3ItaPXFO7ZImXqcW RuKmutNc32nfhTfdTSE+Gn+FP0hyvVehIb12ZNHr5rdWDezRIrhU/4qCY2koq02vv9hS 0JYsKwzPEkKLeHC0o1yMLr4lNeC7nIidqjJQlgZ17X3hwG7uMGUGlb0FYDKstybAoxSN PuzGKEISFL71Mt9LOzHTLH19Bi/ikZqtSvsTTAUBs/9/pSrhf3mQrJw/Bi+Y2mWPIur/ yzHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=izcdKMTlTPXdg1rZpK9PkNPTeAL5ACc5p0aT6ECn+3o=; b=o6ZHYiIEVAdEmcWED/Q1DPumnqt+AmtIP+E6GlLaehT5TwKQw/1HNmdNU8FZU6hjyO vWfCE9xwo3mu4ajGROsj1bMasZS9IvyKFODajiVrrti1pMjSRgfcjDqujYXRb98zptr0 bUfUfKFtGlSDQOawCMuG/Tee6/SzTg77qQN/NaULh5NomQCdb1V6clkkARENuZjboZ1K Nf6KiinbQDZ2pv4v8MFNpL8C0h7WIyvhefp+ZSYRIj0EPjxgL9xGgtF2R0HPRTdDI91t f9bB0DUvjdUk7UNfVDTnb+BrzfNEmGAxtW+TmFGmUbtb231Dq5SP1ow+VIz9x3vHhNGn 1rrA== X-Gm-Message-State: APjAAAUoHG+LlvxR8XDQexFmLZQhMSFRd/Ywe8cMXGWQjEJCzjdc7Zp/ X+c3tTAqScFJb7EZP1H/JA4= X-Google-Smtp-Source: APXvYqwrs3FB89RH0+kgXBF3ejwmYPZLTgHaLNuUiXE1AjcShSZR76+3xPYW28RCBcE0p63294782w== X-Received: by 2002:a17:90a:c214:: with SMTP id e20mr5758828pjt.81.1570040889544; Wed, 02 Oct 2019 11:28:09 -0700 (PDT) Received: from [10.69.78.41] ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id b14sm162486pfi.95.2019.10.02.11.28.07 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 02 Oct 2019 11:28:08 -0700 (PDT) To: Robin Murphy , Nicolas Saenz Julienne , Rob Herring References: <20190924181244.7159-1-nsaenzjulienne@suse.de> <43fb5fe1de317d65a4edf592f88ea150c6e3b8cc.camel@suse.de> <307b988d0c67fb1c42166eca12742bcfda09d92d.camel@suse.de> From: Florian Fainelli Message-ID: Date: Wed, 2 Oct 2019 11:28:06 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.1.1 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Subject: Re: [Xen-devel] [PATCH 00/11] of: Fix DMA configuration for non-DT masters X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: devicetree@vger.kernel.org, Matthias Brugger , linux-arm-msm , linux-wireless , "linux-kernel@vger.kernel.org" , dri-devel , etnaviv@lists.freedesktop.org, linux-tegra@vger.kernel.org, Stefan Wahren , james.quinlan@broadcom.com, linux-pci@vger.kernel.org, "open list:DMA GENERIC OFFLOAD ENGINE SUBSYSTEM" , xen-devel@lists.xenproject.org, Dan Williams , freedreno , Frank Rowand , "moderated list:ARM/FREESCALE IMX / MXC ARM ARCHITECTURE" , Linux Media Mailing List Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" CgpPbiA5LzI2LzIwMTkgNDoyMCBBTSwgUm9iaW4gTXVycGh5IHdyb3RlOgo+IE9uIDIwMTktMDkt MjYgMTE6NDQgYW0sIE5pY29sYXMgU2FlbnogSnVsaWVubmUgd3JvdGU6Cj4+Pj4+PiBSb2Jpbiwg aGF2ZSB5b3UgbG9va2VkIGludG8gc3VwcG9ydGluZyBtdWx0aXBsZSBkbWEtcmFuZ2VzPyBJdCdz IHRoZQo+Pj4+Pj4gbmV4dCB0aGluZwo+Pj4+Pj4gd2UgbmVlZCBmb3IgQkNNIFNUQidzIFBDSWUu IEknbGwgaGF2ZSBhIGdvIGF0IGl0IG15c2VsZiBpZiBub3RoaW5nCj4+Pj4+PiBpcyBpbgo+Pj4+ Pj4gdGhlCj4+Pj4+PiB3b3JrcyBhbHJlYWR5Lgo+Pj4+Pgo+Pj4+PiBNdWx0aXBsZSBkbWEtcmFu Z2VzIGFzIGZhciBhcyBjb25maWd1cmluZyBpbmJvdW5kIHdpbmRvd3Mgc2hvdWxkIHdvcmsKPj4+ Pj4gYWxyZWFkeSBvdGhlciB0aGFuIHRoZSBidWcgd2hlbiB0aGVyZSdzIGFueSBwYXJlbnQgdHJh bnNsYXRpb24uIEJ1dCBpZgo+Pj4+PiB5b3UgbWVhbiBzdXBwb3J0aW5nIG11bHRpcGxlIERNQSBv ZmZzZXRzIGFuZCBtYXNrcyBwZXIgZGV2aWNlIGluIHRoZQo+Pj4+PiBETUEgQVBJLCB0aGVyZSdz IG5vdGhpbmcgaW4gdGhlIHdvcmtzIHlldC4KPj4KPj4gU29ycnksIEkgbWVhbnQgc3VwcG9ydGlu ZyBtdWx0aXBsZSBETUEgb2Zmc2V0c1sxXS4gSSB0aGluayBJIGNvdWxkCj4+IHN0aWxsIG1ha2UK Pj4gaXQgd2l0aCBhIHNpbmdsZSBETUEgbWFzayB0aG91Z2guCj4gCj4gVGhlIG1haW4gcHJvYmxl bSBmb3Igc3VwcG9ydGluZyB0aGF0IGNhc2UgaW4gZ2VuZXJhbCBpcyB0aGUgZGlzZ3VzdGluZwo+ IGNhcnZpbmcgdXAgb2YgdGhlIHBoeXNpY2FsIG1lbW9yeSBtYXAgeW91IG1heSBoYXZlIHRvIGRv IHRvIGd1YXJhbnRlZQo+IHRoYXQgYSBzaW5nbGUgYnVmZmVyIGFsbG9jYXRpb24gY2Fubm90IGV2 ZXIgc3BhbiB0d28gd2luZG93cyB3aXRoCj4gZGlmZmVyZW50IG9mZnNldHMuIEkgZG9uJ3QgdGhp bmsgd2UgZXZlciByZWFjaGVkIGEgY29uY2x1c2lvbiBvbiB3aGV0aGVyCj4gdGhhdCB3YXMgZXZl biBhY2hpZXZhYmxlIGluIHByYWN0aWNlLgoKSXQgaXMgd2l0aCB0aGUgQnJvYWRjb20gU1RCIFNv Q3Mgd2hpY2ggaGF2ZSBiZXR3ZWVuIDEgYW5kIDMgbWVtb3J5CmNvbnRyb2xsZXJzIGRlcGVuZGlu ZyBvbiB0aGUgU29DLCBhbmQgbXVsdGlwbGUgZG1hLXJhbmdlcyBjZWxscyBmb3IgUENJZQphcyBh IGNvbnNlcXVlbmNlLgoKRWFjaCBtZW1vcnkgY29udHJvbGxlciBoYXMgYSBkaWZmZXJlbnQgcGh5 c2ljYWwgYWRkcmVzcyBhcGVydHVyZSBpbiB0aGUKQ1BVJ3MgcGh5c2ljYWwgYWRkcmVzcyBtYXAg KGUuZy46IE1FTUMwIGlzIDB4MCAtIDB4M2ZmZl9mZmZmLCBNRU1DMQoweDQwMDBfMDAwMCAtIDB4 N2ZmZmZfZmZmZiBhbmQgTUVNQzIgMHg4MDAwXzAwMDAgLSAweGJmZmZfZmZmZiwgbm90CmNvdW50 aW5nIHRoZSBleHRlbnNpb24gcmVnaW9ucyBhYm92ZSA0R0IpLCBhbmQgd2hpbGUgdGhlIENQVSBp cwpzY2hlZHVsZWQgYW5kIGFyYml0cmF0ZWQgdGhlIHNhbWUgd2F5IGFjcm9zcyBhbGwgbWVtb3J5 IGNvbnRyb2xsZXJzCih0aHVzIG1ha2luZyBpdCB2aXJ0dWFsbHkgVU1BLCBhbG1vc3QpIGhhdmlu ZyBhIGJ1ZmZlciBzcGFuIHR3byBtZW1vcnkKY29udHJvbGxlcnMgd291bGQgYmUgcHJvYmxlbWF0 aWMgYmVjYXVzZSB0aGUgbWVtb3J5IGNvbnRyb2xsZXJzIGRvIG5vdAprbm93IGhvdyB0byBndWFy YW50ZWUgdGhlIHRyYW5zYWN0aW9uIG9yZGVyaW5nIGFuZCBidWZmZXIgZGF0YQpjb25zaXN0ZW5j eSBpbiBib3RoIERSQU0gaXRzZWxmIGFuZCBmb3Igb3RoZXIgbWVtb3J5IGNvbnRyb2xsZXIgY2xp ZW50cywKbGlrZSBQQ0llLgoKV2UgaGlzdG9yaWNhbGx5IGhhZCB0byByZXNlcnZlIHRoZSBsYXN0 IDRLQiBvZiBlYWNoIG1lbW9yeSBjb250cm9sbGVyIHRvCmF2b2lkIHByb2JsZW1hdGljIGNvbnRy b2xsZXJzIGxpa2UgRUhDSSB0byBwcmVmZXRjaCBiZXlvbmQgdGhlIGVuZCBvZiBhCm1lbW9yeSBj b250cm9sbGVyJ3MgcG9wdWxhdGVkIG1lbW9yeSBhbmQgdGhhdCBhbHNvIGluY2lkZW50YWxseSB0 YWtlcwpjYXJlIG9mIG5ldmVyIGhhdmluZyBhIGJ1ZmZlciBjcm9zcyBhIGNvbnRyb2xsZXIgYm91 bmRhcnkuIEVpdGhlciB5b3UKY2FuIGFsbG9jYXRlIHRoZSBlbnRpcmUgYnVmZmVyIG9uIGEgZ2l2 ZW4gbWVtb3J5IGNvbnRyb2xsZXIsIG9yIHlvdQpjYW5ub3QgYWxsb2NhdGUgbWVtb3J5IGF0IGFs bCBvbiB0aGF0IHpvbmUvcmVnaW9uIGFuZCBhbm90aGVyIG9uZSBtdXN0CmJlIGZvdW5kIChvciB0 aGVyZSBpcyBubyBtb3JlIG1lbW9yeSBhbmQgdGhlcmUgaXMgYSBnZW51aW5lIE9PTSkuCgpUaGUg d2F5IHdlIHJlc2VydmUgbWVtb3J5IHJpZ2h0IG5vdyBpcyBiYXNlZCBvbiB0aGUgZmlyc3QgcGF0 Y2gKc3VibWl0dGVkIGJ5IEppbToKCmh0dHBzOi8vbG9yZS5rZXJuZWwub3JnL3BhdGNod29yay9w YXRjaC85ODg0NjkvCgp3aGVyZWJ5IHdlIHJlYWQgdGhlIG1lbW9yeSBub2RlJ3MgInJlZyIgcHJv cGVydHkgYW5kIHdlIG1hcCB0aGUgcGh5c2ljYWwKYWRkcmVzc2VzIHRvIHRoZSBtZW1vcnkgY29u dHJvbGxlciBjb25maWd1cmF0aW9uIHJlYWQgZnJvbSB0aGUgc3BlY2lmaWMKcmVnaXN0ZXJzIGlu IHRoZSBDUFUncyBCdXMgSW50ZXJmYWNlIFVuaXQgKHdoZXJlIHRoZSBtZW1vcnkgY29udHJvbGxl cgphcGVydHVyZXMgYXJlIGFyY2hpdGVjdHVyYWxseSBkZWZpbmVkKSBhbmQgdGhlbiB3ZSB1c2Ug dGhhdCB0byBjYWxsCm1lbWJsb2NrX3Jlc2VydmUoKSAobm90IHBhcnQgb2YgdGhhdCBwYXRjaCwg aXQgc2hvdWxkIGJlIHRob3VnaCkuCi0tIApGbG9yaWFuCgpfX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fXwpYZW4tZGV2ZWwgbWFpbGluZyBsaXN0Clhlbi1kZXZl bEBsaXN0cy54ZW5wcm9qZWN0Lm9yZwpodHRwczovL2xpc3RzLnhlbnByb2plY3Qub3JnL21haWxt YW4vbGlzdGluZm8veGVuLWRldmVs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.2 required=3.0 tests=DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BADDC352AA for ; Wed, 2 Oct 2019 18:28:18 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 65135218DE for ; Wed, 2 Oct 2019 18:28:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Ib2SdKPm"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nr6ZKcXN" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 65135218DE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date: Message-ID:From:References:To:Subject:Reply-To:Content-ID:Content-Description :Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=N/on1c7LmQOzTtOaJd5pPJTn0bKtvHuDXDA3cmQQWh4=; b=Ib2SdKPmSnSW+l Lv4XwQb84MvbEAA2hnVD01F4QX9TI0evPSpGQnUPjg485Pf9+ACc5w0yEqi2eYzHbsQq9yVBVy5YK TMjK9SQ55oK02cvHDjGWrFIhr4BlVdwq40CF7B+WXqigZFEGpPsz7Rzh3PAUu9ySLNh/hJAMM1lPi A5zi7tVdb2m9kg61YSXKPJ/okCmX1PgIe3PC1qUv9MgVDbEQCnTCTOdMNYxGP8NZcCAKSGRnSGQ6A FFB+1T1LNzMgsV7bjkG2fYjvnYCENqXq/ItIuGvAjBmJMLFZRmL0gQ/YD7icFZZrQWK34I1PyLos8 m60pCKlFPZ9RoBaSxGVQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.2 #3 (Red Hat Linux)) id 1iFjMD-0004id-U8; Wed, 02 Oct 2019 18:28:13 +0000 Received: from mail-pg1-x542.google.com ([2607:f8b0:4864:20::542]) by bombadil.infradead.org with esmtps (Exim 4.92.2 #3 (Red Hat Linux)) id 1iFjMA-0004i7-JF for linux-arm-kernel@lists.infradead.org; Wed, 02 Oct 2019 18:28:12 +0000 Received: by mail-pg1-x542.google.com with SMTP id e1so6829953pgj.6 for ; Wed, 02 Oct 2019 11:28:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=izcdKMTlTPXdg1rZpK9PkNPTeAL5ACc5p0aT6ECn+3o=; b=nr6ZKcXN/729CI5YvKkqVc0Bz2wyp7AqMiNJbnneIbJ2S8a6/RpgT+vQxveooa1g8o Tzstjrebdj0ch6CBEQIhLHedt+Meqh8poQODSN2lg0OykjXXZCvv3ItaPXFO7ZImXqcW RuKmutNc32nfhTfdTSE+Gn+FP0hyvVehIb12ZNHr5rdWDezRIrhU/4qCY2koq02vv9hS 0JYsKwzPEkKLeHC0o1yMLr4lNeC7nIidqjJQlgZ17X3hwG7uMGUGlb0FYDKstybAoxSN PuzGKEISFL71Mt9LOzHTLH19Bi/ikZqtSvsTTAUBs/9/pSrhf3mQrJw/Bi+Y2mWPIur/ yzHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=izcdKMTlTPXdg1rZpK9PkNPTeAL5ACc5p0aT6ECn+3o=; b=iePFNzIdsv5HktV2Hk1AO5i+KXWCm1E3huy3nLMy9NS2n8BNK0J5SXA4m1wzzkuLsW 01sCF7EankP5c3dY702jMf4Nw3G6MaJqrcrdeT9UXWQ8ES8RRMcljL8guLKMhDKZbe1y E1j1uhNLPmWJ5pu5mV4cm/h0ZQc7KDwCpX+BxsneKXTYwPGCjvBHPJ1ojbA6GUsQKz3o qPGvz28JQRA5VaC6iIM1JPp5WMGKsJ/DY3MoZ5ilkm7jDLmg4NkaZfaRjqcByZVqJNhN hVFJrdlAZ78AfAbAGnXqLs4diTxY0XnOZpcyOSmCYsmrvAPMFAL5+KaUwmxlMHJIMIcJ 81VQ== X-Gm-Message-State: APjAAAV0yKXuj3WMWsEKEn0bU/EMAWpGSYa1WIFrozISuW6TSIHTzNoh lBeqUcOQomhQSBXiYQY3uPA= X-Google-Smtp-Source: APXvYqwrs3FB89RH0+kgXBF3ejwmYPZLTgHaLNuUiXE1AjcShSZR76+3xPYW28RCBcE0p63294782w== X-Received: by 2002:a17:90a:c214:: with SMTP id e20mr5758828pjt.81.1570040889544; Wed, 02 Oct 2019 11:28:09 -0700 (PDT) Received: from [10.69.78.41] ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id b14sm162486pfi.95.2019.10.02.11.28.07 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 02 Oct 2019 11:28:08 -0700 (PDT) Subject: Re: [PATCH 00/11] of: Fix DMA configuration for non-DT masters To: Robin Murphy , Nicolas Saenz Julienne , Rob Herring References: <20190924181244.7159-1-nsaenzjulienne@suse.de> <43fb5fe1de317d65a4edf592f88ea150c6e3b8cc.camel@suse.de> <307b988d0c67fb1c42166eca12742bcfda09d92d.camel@suse.de> From: Florian Fainelli Message-ID: Date: Wed, 2 Oct 2019 11:28:06 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.1.1 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191002_112810_633605_1A9B9EB4 X-CRM114-Status: GOOD ( 19.09 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: devicetree@vger.kernel.org, Matthias Brugger , linux-arm-msm , linux-wireless , "linux-kernel@vger.kernel.org" , dri-devel , etnaviv@lists.freedesktop.org, linux-tegra@vger.kernel.org, Stefan Wahren , james.quinlan@broadcom.com, linux-pci@vger.kernel.org, "open list:DMA GENERIC OFFLOAD ENGINE SUBSYSTEM" , xen-devel@lists.xenproject.org, Dan Williams , freedreno , Frank Rowand , "moderated list:ARM/FREESCALE IMX / MXC ARM ARCHITECTURE" , Linux Media Mailing List Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 9/26/2019 4:20 AM, Robin Murphy wrote: > On 2019-09-26 11:44 am, Nicolas Saenz Julienne wrote: >>>>>> Robin, have you looked into supporting multiple dma-ranges? It's the >>>>>> next thing >>>>>> we need for BCM STB's PCIe. I'll have a go at it myself if nothing >>>>>> is in >>>>>> the >>>>>> works already. >>>>> >>>>> Multiple dma-ranges as far as configuring inbound windows should work >>>>> already other than the bug when there's any parent translation. But if >>>>> you mean supporting multiple DMA offsets and masks per device in the >>>>> DMA API, there's nothing in the works yet. >> >> Sorry, I meant supporting multiple DMA offsets[1]. I think I could >> still make >> it with a single DMA mask though. > > The main problem for supporting that case in general is the disgusting > carving up of the physical memory map you may have to do to guarantee > that a single buffer allocation cannot ever span two windows with > different offsets. I don't think we ever reached a conclusion on whether > that was even achievable in practice. It is with the Broadcom STB SoCs which have between 1 and 3 memory controllers depending on the SoC, and multiple dma-ranges cells for PCIe as a consequence. Each memory controller has a different physical address aperture in the CPU's physical address map (e.g.: MEMC0 is 0x0 - 0x3fff_ffff, MEMC1 0x4000_0000 - 0x7ffff_ffff and MEMC2 0x8000_0000 - 0xbfff_ffff, not counting the extension regions above 4GB), and while the CPU is scheduled and arbitrated the same way across all memory controllers (thus making it virtually UMA, almost) having a buffer span two memory controllers would be problematic because the memory controllers do not know how to guarantee the transaction ordering and buffer data consistency in both DRAM itself and for other memory controller clients, like PCIe. We historically had to reserve the last 4KB of each memory controller to avoid problematic controllers like EHCI to prefetch beyond the end of a memory controller's populated memory and that also incidentally takes care of never having a buffer cross a controller boundary. Either you can allocate the entire buffer on a given memory controller, or you cannot allocate memory at all on that zone/region and another one must be found (or there is no more memory and there is a genuine OOM). The way we reserve memory right now is based on the first patch submitted by Jim: https://lore.kernel.org/patchwork/patch/988469/ whereby we read the memory node's "reg" property and we map the physical addresses to the memory controller configuration read from the specific registers in the CPU's Bus Interface Unit (where the memory controller apertures are architecturally defined) and then we use that to call memblock_reserve() (not part of that patch, it should be though). -- Florian _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel