From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Subject: [v1] dmaengine: tegra: Use relaxed versions of readl/writel From: Dmitry Osipenko Message-Id: Date: Fri, 26 Apr 2019 16:03:08 +0300 To: Jon Hunter , Laxman Dewangan , Vinod Koul , Thierry Reding Cc: dmaengine@vger.kernel.org, linux-tegra@vger.kernel.org, linux-kernel@vger.kernel.org List-ID: MjYuMDQuMjAxOSAxNTo0MiwgRG1pdHJ5IE9zaXBlbmtvINC/0LjRiNC10YI6Cj4gMjYuMDQuMjAx OSAxNToxOCwgRG1pdHJ5IE9zaXBlbmtvINC/0LjRiNC10YI6Cj4+IDI2LjA0LjIwMTkgMTQ6MTMs IEpvbiBIdW50ZXIg0L/QuNGI0LXRgjoKPj4+Cj4+PiBPbiAyNi8wNC8yMDE5IDExOjQ1LCBEbWl0 cnkgT3NpcGVua28gd3JvdGU6Cj4+Pj4gMjYuMDQuMjAxOSAxMjo1MiwgSm9uIEh1bnRlciDQv9C4 0YjQtdGCOgo+Pj4+Pgo+Pj4+PiBPbiAyNS8wNC8yMDE5IDAwOjE3LCBEbWl0cnkgT3NpcGVua28g d3JvdGU6Cj4+Pj4+PiBUaGUgcmVhZGwvd3JpdGVsIGZ1bmN0aW9ucyBhcmUgaW5zZXJ0aW5nIG1l bW9yeSBiYXJyaWVyIGluIG9yZGVyIHRvCj4+Pj4+PiBlbnN1cmUgdGhhdCBtZW1vcnkgc3RvcmVz IGFyZSBjb21wbGV0ZWQuIE9uIFRlZ3JhMjAgYW5kIFRlZ3JhMzAgdGhpcwo+Pj4+Pj4gcmVzdWx0 cyBpbiBMMiBjYWNoZSBzeW5jaW5nIHdoaWNoIGlzbid0IGEgY2hlYXBlc3Qgb3BlcmF0aW9uLiBU aGUKPj4+Pj4+IHRlZ3JhMjAtYXBiLWRtYSBkcml2ZXIgZG9lc24ndCBuZWVkIHRvIHN5bmNocm9u aXplIGdlbmVyaWMgbWVtb3J5Cj4+Pj4+PiBhY2Nlc3NlcywgaGVuY2UgdXNlIHRoZSByZWxheGVk IHZlcnNpb25zIG9mIHRoZSBmdW5jdGlvbnMuCj4+Pj4+Cj4+Pj4+IERvIHlvdSBtZWFuIGRldmlj ZS1pbyBhY2Nlc3NlcyBoZXJlIGFzIHRoaXMgaXMgbm90IGdlbmVyaWMgbWVtb3J5Pwo+Pj4+Cj4+ Pj4gWWVzLiBUaGUgSU9NRU0gYWNjZXNzZXMgd2l0aGluIGFyZSBhbHdheXMgb3JkZXJlZCBhbmQg dW5jYWNoZWQsIHdoaWxlCj4+Pj4gZ2VuZXJpYyBtZW1vcnkgYWNjZXNzZXMgYXJlIG91dC1vZi1v cmRlciBhbmQgY2FjaGVkLgo+Pj4+Cj4+Pj4+IEFsdGhvdWdoIHRoZXJlIG1heSBub3QgYmUgYW55 IGlzc3VlcyB3aXRoIHRoaXMgY2hhbmdlLCBJIHRoaW5rIEkgbmVlZCBhCj4+Pj4+IGJpdCBtb3Jl IGNvbnZpbmNpbmcgdGhhdCB3ZSBzaG91bGQgZG8gdGhpcyBnaXZlbiB0aGF0IHdlIGhhdmUgaGFk IGl0Cj4+Pj4+IHRoaXMgd2F5IGZvciBzb21ldGltZSBhbmQgSSB3b3VsZCBub3QgbGlrZSB0byBz ZWUgdXMgaW50cm9kdWNlIGFueQo+Pj4+PiByZWdyZXNzaW9ucyBhcyB0aGlzIHBvaW50IHdpdGhv dXQgYmVpbmcgMTAwJSBjZXJ0YWluIHdlIHdvdWxkIG5vdC4KPj4+Pj4gSWRlYWxseSwgaWYgSSBo YWQgc29tZSBnb29kIGV4dGVuc2l2ZSB0ZXN0cyBJIGNvdWxkIHJ1biB0byBoYW1tZXIgdGhlCj4+ Pj4+IERNQSBmb3IgYWxsIGNvbmZpZ3VyYXRpb25zIHdpdGggZGlmZmVyZW50IGNvbWJpbmF0aW9u cyBvZiBjaGFubmVscwo+Pj4+PiBydW5uaW5nIHNpbXVsdGFuZW91c2x5IHRoZW4gd2UgY291bGQg dGVzdCB0aGlzLCBidXQgcmlnaHQgbm93IEkgZG9uJ3QgOi0oCj4+Pj4+Cj4+Pj4+IEhhdmUgeW91 IC4uLgo+Pj4+PiAxLiBUZXN0ZWQgYm90aCBjeWNsaWMgYW5kIHNjYXR0ZXItZ2F0aGVyIHRyYW5z ZmVycz8KPj4+Pj4gMi4gU3RyZXNzIHRlc3RlZCBzaW11bHRhbmVvdXMgdHJhbnNmZXJzIHdpdGgg dmFyaW91cyBkaWZmZXJlbnQKPj4+Pj4gICAgY29uZmlndXJhdGlvbnM/Cj4+Pj4+IDMuIFF1YW50 aWZpZWQgdGhlIGFjdHVhbCBwZXJmb3JtYW5jZSBiZW5lZml0IG9mIHRoaXMgY2hhbmdlIHNvIHdl IGNhbgo+Pj4+PiAgICB1bmRlcnN0YW5kIGhvdyBtdWNoIG9mIGEgcGVyZm9ybWFuY2UgYm9vc3Qg dGhpcyBvZmZlcnM/Cj4+Pj4KPj4+PiBBY3R1YWxseSBJIGZvdW5kIGEgY2FzZSB3aGVyZSB0aGlz IGNoYW5nZSBjYXVzZXMgYSBwcm9ibGVtLCBJJ20gc2VlaW5nCj4+Pj4gSTJDIHRyYW5zZmVyIHRp bWVvdXQgZm9yIHRvdWNoc2NyZWVuIGFuZCBpdCBicmVha3MgdGhlIHRvdWNoIGlucHV0Lgo+Pj4+ IEluZGVlZCwgSSBoYXZlbid0IHRlc3RlZCB0aGlzIHBhdGNoIHZlcnkgd2VsbC4KPj4+Pgo+Pj4+ IEFuZCB0aGUgZml4IGlzIHRoaXM6Cj4+Pj4KPj4+PiBAQCAtMTU5Miw2ICsxNTkyLDggQEAgc3Rh dGljIGludCB0ZWdyYV9kbWFfcnVudGltZV9zdXNwZW5kKHN0cnVjdCBkZXZpY2UKPj4+PiAqZGV2 KQo+Pj4+ICAJCQkJCQkgIFRFR1JBX0FQQkRNQV9DSEFOX1dDT1VOVCk7Cj4+Pj4gIAl9Cj4+Pj4K Pj4+PiArCWRzYigpOwo+Pj4+ICsKPj4+PiAgCWNsa19kaXNhYmxlX3VucHJlcGFyZSh0ZG1hLT5k bWFfY2xrKTsKPj4+Pgo+Pj4+ICAJcmV0dXJuIDA7Cj4+Pj4KPj4+Pgo+Pj4+IEFwcGFyZW50bHkg dGhlIHByb2JsZW0gaXMgdGhhdCBDTEsvRE1BIChQUFNCL0FQQikgYWNjZXNzZXMgYXJlCj4+Pj4g aW5jb2hlcmVudCBhbmQgQ1BVIGRpc2FibGVzIGNsb2NrIGJlZm9yZSB3cml0ZXMgYXJlIHJlYWNo aW5nIERNQSBjb250cm9sbGVyLgo+Pj4+Cj4+Pj4gSSdkIHNheSB0aGF0IGN5Y2xpYyBhbmQgc2Nh dHRlci1nYXRoZXIgdHJhbnNmZXJzIGFyZSBub3cgdGVzdGVkLiBJIGFsc28KPj4+PiBtYWRlIHNv bWUgbW9yZSB0ZXN0aW5nIG9mIHNpbXVsdGFuZW91cyB0cmFuc2ZlcnMuCj4+Pj4KPj4+PiBRdWFu dGlmeWluZyBwZXJmb3JtYW5jZSBwcm9iYWJseSB3b24ndCBiZSBlYXN5IHRvIG1ha2UgYXMgdGhl IERNQQo+Pj4+IHJlYWQvd3JpdGVzIGFyZSBub3Qgb24gYW55IGtpbmQgb2YgY29kZSdzIGhvdC1w YXRoLgo+Pj4KPj4+IFNvIHdoeSBtYWtlIHRoZSBjaGFuZ2U/Cj4+Cj4+IEZvciBjb25zaXN0ZW5j eS4KPj4KPj4+PiBKb24sIGFyZSB5b3Ugc3RpbGwgaW5zaXN0aW5nIGFib3V0IHRvIGRyb3AgdGhp cyBwYXRjaCBvciB5b3Ugd2lsbCBiZQo+Pj4+IGZpbmUgd2l0aCB0aGUgdjIgdGhhdCB3aWxsIGhh dmUgdGhlIGRzYigpIGluIHBsYWNlPwo+Pj4KPj4+IElmIHdlIGNhbid0IHF1YW50aWZ5IHRoZSBw ZXJmb3JtYW5jZSBnYWluLCB0aGVuIGl0IGlzIGRpZmZpY3VsdCB0bwo+Pj4ganVzdGlmeSB0aGUg Y2hhbmdlLiBJIHdvdWxkIGFsc28gYmUgY29uY2VybmVkIGlmIHRoYXQgaXMgdGhlIG9ubHkgcGxh Y2UKPj4+IHdlIG5lZWQgYW4gZXhwbGljaXQgZHNiLgo+Pgo+PiBNYXliZSBpdCB3b24ndCBodXJ0 IHRvIGFkZCBkc2IgdG8gdGhlIElTUiBhcyB3ZWxsLiBCdXQgb2theSwgbGV0J3MgZHJvcAo+PiB0 aGlzIHBhdGNoIGZvciBub3cuCj4+Cj4gCj4gSm9uLCBpdCBvY2N1cnJlZCB0byBtZSB0aGF0IHRo ZXJlIHN0aWxsIHNob3VsZCBiZSBhIHByb2JsZW0gd2l0aCB0aGUKPiB3cml0ZWwoKSBvcmRlcmlu ZyBpbiB0aGUgZHJpdmVyIGJlY2F1c2Ugd3JpdGVsKCkgZW5zdXJlcyB0aGF0IG1lbW9yeQo+IHN0 b3JlcyBhcmUgY29tcGxldGVkICpiZWZvcmUqIHRoZSB3cml0ZSBvY2N1cnMgYW5kIGhlbmNlIHRy YW5zbGF0ZXMgaW50bwo+IGlvd21iKCkgKyB3cml0ZWxfcmVsYXhlZCgpIFswXS4gVGh1cyB0aGUg bGFzdCB3cml0ZSB3aWxsIGFsd2F5cyBoYXBwZW4KPiBhc3luY2hyb25vdXNseSBpbiByZWdhcmRz IHRvIGNsayBhY2Nlc3Nlcy4KPiAKPiBbMF0KPiBodHRwczovL2dpdC5rZXJuZWwub3JnL3B1Yi9z Y20vbGludXgva2VybmVsL2dpdC9uZXh0L2xpbnV4LW5leHQuZ2l0L3RyZWUvYXJjaC9hcm0vaW5j bHVkZS9hc20vaW8uaCNuMzExCj4gCgpBbHNvIHBsZWFzZSBub3RlIHRoYXQgaW93bWIoKSB0cmFu c2xhdGVzIGludG8gd21iKCkgaWYKQ09ORklHX0FSTV9ETUFfTUVNX0JVRkZFUkFCTEU9eSBhbmQg c29tZXRpbWUgYWdvIEkgd2FzIHByb2ZpbGluZyBob3N0MXgKZHJpdmVyIGpvYiBzdWJtaXNzaW9u IHBlcmZvcm1hbmNlIGFuZCBoYXZlIHNlZW4gY2FzZXMgd2hlcmUgd21iKCkgY291bGQKdGFrZSB1 cCB0byAxbXMgb24gVDIwIGR1ZSB0byBMMiBzeW5jaW5nIGlmIHRoZXJlIGFyZSBvdXRzdGFuZGlu ZyBtZW1vcnkKd3JpdGVzIGluIHRoZSBjYWNoZSAob3IgZXZlbiBtb3JlLCBJIGRvbid0IHJlbWVt YmVyIGV4YWN0bHkgYWxyZWFkeSBob3cKYmFkIGl0IHdhcy4uKS4KCkFsdG9nZXRoZXIsIEkgdGhp bmsgdGhlIHVzYWdlIG9mIHJlYWRsL3dyaXRlbCBpbiBwcmV0dHkgbXVjaCBhbGwgb2YKVGVncmEg ZHJpdmVycyBpcyBwbGFpbmx5IHdyb25nIGFuZCBleHBsaWNpdCBkc2IoKSBzaGFsbCBiZSB1c2Vk IGluCnBsYWNlcyB3aGVyZSBoYXJkd2FyZSBzeW5jaHJvbml6YXRpb24gaXMgcmVhbGx5IG5lZWRl ZC4K From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.0 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AB7DC43219 for ; Fri, 26 Apr 2019 13:03:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1D4082067D for ; Fri, 26 Apr 2019 13:03:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MHPQrblC" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726197AbfDZNDN (ORCPT ); Fri, 26 Apr 2019 09:03:13 -0400 Received: from mail-lf1-f67.google.com ([209.85.167.67]:38773 "EHLO mail-lf1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726308AbfDZNDN (ORCPT ); Fri, 26 Apr 2019 09:03:13 -0400 Received: by mail-lf1-f67.google.com with SMTP id v1so2427828lfg.5; Fri, 26 Apr 2019 06:03:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:references:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=eQoi9xbGcMcYfOoWsGLbOHVHNTXp7aikInN/pNRji8Q=; b=MHPQrblCLmuF8oN4myJ4270iLPUopYcgS+X75r85ziPmgMpd8OP8ONi+DRskxigUcG JhdwrDOPT7UP8clNTvUld3qEusk3DEu/w3+S852X1TT78f03Fferiyhxej6x01q6vU5q Noau5yF2VFx0dSOkkTUiHTGgRrVa5QYoQ57ykmYdbcvGlVXlZ16VZ6f9hq3gKyqe+99w Y9sTuXrh1VBi3lgM1zOTfx4oLLel8BZP0Gp+bzEZXSnt5xvY9ehEaKBfjwMRBvehN3QO hBPHwe10vO7H9QUVBao46+qtlnmAi+s/M/QTUXhCzIpRuWAg6cOEifO7YUtGqlFDeuHB aIeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:references:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=eQoi9xbGcMcYfOoWsGLbOHVHNTXp7aikInN/pNRji8Q=; b=niJsP9zDCamuh4HlbAT1i4ZVOlrtAqv6eEUOoBbyzarROXg9qjpi5rggSIA65tWRsG Al8DRULy0EmCagZpXdLuqKq3/HqK0mIsDN1tN5GnkRkAQEzRXzwBiO47V1wSP0p/7eQf UcBSNClONnQHfSJSD+ZEk+oYwFJ0Er2qWvtpAdNqgb+zkjPEisyAzrTYeMPbwj4fM0eR KlPsRC/rnvW11cHE3jzM5bKc03uMZ32Y4WulC1RSCmpJd+OtECGVPAEy+VDPBlshFEAY 4sikvEa473vvTyMtPLueNa5m1tioz3QEYNHa3jb6toXbMNsbqW0Uox6Da3r8PP8bTXlC Vz3w== X-Gm-Message-State: APjAAAUhIoxUXPm0RM6HJFHeJ3cY8NRIfp7API7eh16yuHWyAizsyLfK +1idsvRCdAlAMF/GO56xUDH0CNBo X-Google-Smtp-Source: APXvYqxGr9IvMFAvCl6D/qvGYxV5I2Gx8U81/nb+rVq5F67+UwtbSead7yVV4X5EjlovziWV7fWxRw== X-Received: by 2002:ac2:558d:: with SMTP id v13mr17676257lfg.76.1556283790000; Fri, 26 Apr 2019 06:03:10 -0700 (PDT) Received: from [192.168.2.145] (ppp94-29-35-107.pppoe.spdop.ru. [94.29.35.107]) by smtp.googlemail.com with ESMTPSA id v28sm5564779lfi.33.2019.04.26.06.03.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 06:03:09 -0700 (PDT) Subject: Re: [PATCH v1] dmaengine: tegra: Use relaxed versions of readl/writel From: Dmitry Osipenko To: Jon Hunter , Laxman Dewangan , Vinod Koul , Thierry Reding Cc: dmaengine@vger.kernel.org, linux-tegra@vger.kernel.org, linux-kernel@vger.kernel.org References: <20190424231708.21219-1-digetx@gmail.com> <4a315b63-bc71-3c3e-f1ae-8638bcf4033d@gmail.com> <49392c02-6dcc-9a95-0035-27c4c0d14820@gmail.com> <242863b9-b75e-4b37-178a-5aa03e56d3e1@gmail.com> Message-ID: Date: Fri, 26 Apr 2019 16:03:08 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <242863b9-b75e-4b37-178a-5aa03e56d3e1@gmail.com> Content-Type: text/plain; charset="UTF-8" Content-Language: en-US Content-Transfer-Encoding: 8bit Sender: dmaengine-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org Message-ID: <20190426130308.kN8o8SoRoimFc-IcS_uFwqAtXXySTYqAx-XO2MNU6-U@z> 26.04.2019 15:42, Dmitry Osipenko пишет: > 26.04.2019 15:18, Dmitry Osipenko пишет: >> 26.04.2019 14:13, Jon Hunter пишет: >>> >>> On 26/04/2019 11:45, Dmitry Osipenko wrote: >>>> 26.04.2019 12:52, Jon Hunter пишет: >>>>> >>>>> On 25/04/2019 00:17, Dmitry Osipenko wrote: >>>>>> The readl/writel functions are inserting memory barrier in order to >>>>>> ensure that memory stores are completed. On Tegra20 and Tegra30 this >>>>>> results in L2 cache syncing which isn't a cheapest operation. The >>>>>> tegra20-apb-dma driver doesn't need to synchronize generic memory >>>>>> accesses, hence use the relaxed versions of the functions. >>>>> >>>>> Do you mean device-io accesses here as this is not generic memory? >>>> >>>> Yes. The IOMEM accesses within are always ordered and uncached, while >>>> generic memory accesses are out-of-order and cached. >>>> >>>>> Although there may not be any issues with this change, I think I need a >>>>> bit more convincing that we should do this given that we have had it >>>>> this way for sometime and I would not like to see us introduce any >>>>> regressions as this point without being 100% certain we would not. >>>>> Ideally, if I had some good extensive tests I could run to hammer the >>>>> DMA for all configurations with different combinations of channels >>>>> running simultaneously then we could test this, but right now I don't :-( >>>>> >>>>> Have you ... >>>>> 1. Tested both cyclic and scatter-gather transfers? >>>>> 2. Stress tested simultaneous transfers with various different >>>>> configurations? >>>>> 3. Quantified the actual performance benefit of this change so we can >>>>> understand how much of a performance boost this offers? >>>> >>>> Actually I found a case where this change causes a problem, I'm seeing >>>> I2C transfer timeout for touchscreen and it breaks the touch input. >>>> Indeed, I haven't tested this patch very well. >>>> >>>> And the fix is this: >>>> >>>> @@ -1592,6 +1592,8 @@ static int tegra_dma_runtime_suspend(struct device >>>> *dev) >>>> TEGRA_APBDMA_CHAN_WCOUNT); >>>> } >>>> >>>> + dsb(); >>>> + >>>> clk_disable_unprepare(tdma->dma_clk); >>>> >>>> return 0; >>>> >>>> >>>> Apparently the problem is that CLK/DMA (PPSB/APB) accesses are >>>> incoherent and CPU disables clock before writes are reaching DMA controller. >>>> >>>> I'd say that cyclic and scatter-gather transfers are now tested. I also >>>> made some more testing of simultaneous transfers. >>>> >>>> Quantifying performance probably won't be easy to make as the DMA >>>> read/writes are not on any kind of code's hot-path. >>> >>> So why make the change? >> >> For consistency. >> >>>> Jon, are you still insisting about to drop this patch or you will be >>>> fine with the v2 that will have the dsb() in place? >>> >>> If we can't quantify the performance gain, then it is difficult to >>> justify the change. I would also be concerned if that is the only place >>> we need an explicit dsb. >> >> Maybe it won't hurt to add dsb to the ISR as well. But okay, let's drop >> this patch for now. >> > > Jon, it occurred to me that there still should be a problem with the > writel() ordering in the driver because writel() ensures that memory > stores are completed *before* the write occurs and hence translates into > iowmb() + writel_relaxed() [0]. Thus the last write will always happen > asynchronously in regards to clk accesses. > > [0] > https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/arch/arm/include/asm/io.h#n311 > Also please note that iowmb() translates into wmb() if CONFIG_ARM_DMA_MEM_BUFFERABLE=y and sometime ago I was profiling host1x driver job submission performance and have seen cases where wmb() could take up to 1ms on T20 due to L2 syncing if there are outstanding memory writes in the cache (or even more, I don't remember exactly already how bad it was..). Altogether, I think the usage of readl/writel in pretty much all of Tegra drivers is plainly wrong and explicit dsb() shall be used in places where hardware synchronization is really needed.