From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eugeniy Paltsev Subject: Re: [PATCH v2 2/4] ARC: allow to use IOC and non-IOC DMA devices simultaneously Date: Mon, 13 Aug 2018 17:08:10 +0000 Message-ID: <1534180089.3962.68.camel@synopsys.com> References: <20180730162636.3556-1-Eugeniy.Paltsev@synopsys.com> <20180730162636.3556-3-Eugeniy.Paltsev@synopsys.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: Content-Language: en-US Content-ID: <5A3D85E4A242D94C9FF74AF83992F416@internal.synopsys.com> Sender: linux-kernel-owner@vger.kernel.org To: "Eugeniy.Paltsev@synopsys.com" , Vineet Gupta , "linux-snps-arc@lists.infradead.org" Cc: "hch@lst.de" , "linux-kernel@vger.kernel.org" , "linux-arch@vger.kernel.org" , Alexey Brodkin List-Id: linux-arch.vger.kernel.org T24gTW9uLCAyMDE4LTA4LTEzIGF0IDE2OjI0ICswMDAwLCBWaW5lZXQgR3VwdGEgd3JvdGU6DQo+ IE9uIDA3LzMwLzIwMTggMDk6MjYgQU0sIEV1Z2VuaXkgUGFsdHNldiB3cm90ZToNCj4gPiBAQCAt MTI2MywxMSArMTI1NCw3IEBAIHZvaWQgX19pbml0IGFyY19jYWNoZV9pbml0X21hc3Rlcih2b2lk KQ0KPiA+ICAJaWYgKGlzX2lzYV9hcmN2MigpICYmIGlvY19lbmFibGUpDQo+ID4gIAkJYXJjX2lv Y19zZXR1cCgpOw0KPiA+ICANCj4gPiAtCWlmIChpc19pc2FfYXJjdjIoKSAmJiBpb2NfZW5hYmxl KSB7DQo+ID4gLQkJX19kbWFfY2FjaGVfd2JhY2tfaW52ID0gX19kbWFfY2FjaGVfd2JhY2tfaW52 X2lvYzsNCj4gPiAtCQlfX2RtYV9jYWNoZV9pbnYgPSBfX2RtYV9jYWNoZV9pbnZfaW9jOw0KPiA+ IC0JCV9fZG1hX2NhY2hlX3diYWNrID0gX19kbWFfY2FjaGVfd2JhY2tfaW9jOw0KPiA+IC0JfSBl bHNlIGlmIChpc19pc2FfYXJjdjIoKSAmJiBsMl9saW5lX3N6ICYmIHNsY19lbmFibGUpIHsNCj4g PiArCWlmIChpc19pc2FfYXJjdjIoKSAmJiBsMl9saW5lX3N6ICYmIHNsY19lbmFibGUpIHsNCj4g PiAgCQlfX2RtYV9jYWNoZV93YmFja19pbnYgPSBfX2RtYV9jYWNoZV93YmFja19pbnZfc2xjOw0K PiA+ICAJCV9fZG1hX2NhY2hlX2ludiA9IF9fZG1hX2NhY2hlX2ludl9zbGM7DQo+ID4gIAkJX19k bWFfY2FjaGVfd2JhY2sgPSBfX2RtYV9jYWNoZV93YmFja19zbGM7DQo+ID4gZGlmZiAtLWdpdCBh L2FyY2gvYXJjL21tL2RtYS5jIGIvYXJjaC9hcmMvbW0vZG1hLmMNCj4gPiBpbmRleCBjZWZiNzc2 YTk5ZmYuLjRkMTQ2NjkwNWU0OCAxMDA2NDQNCj4gPiAtLS0gYS9hcmNoL2FyYy9tbS9kbWEuYw0K PiA+ICsrKyBiL2FyY2gvYXJjL21tL2RtYS5jDQo+ID4gQEAgLTMzLDE5ICszMyw3IEBAIHZvaWQg KmFyY2hfZG1hX2FsbG9jKHN0cnVjdCBkZXZpY2UgKmRldiwgc2l6ZV90IHNpemUsIGRtYV9hZGRy X3QgKmRtYV9oYW5kbGUsDQo+ID4gIAlpZiAoIXBhZ2UpDQo+ID4gIAkJcmV0dXJuIE5VTEw7DQo+ ID4gIA0KPiA+IC0JLyoNCj4gPiAtCSAqIElPQyByZWxpZXMgb24gYWxsIGRhdGEgKGV2ZW4gY29o ZXJlbnQgRE1BIGRhdGEpIGJlaW5nIGluIGNhY2hlDQo+ID4gLQkgKiBUaHVzIGFsbG9jYXRlIG5v cm1hbCBjYWNoZWQgbWVtb3J5DQo+ID4gLQkgKg0KPiA+IC0JICogVGhlIGdhaW5zIHdpdGggSU9D IGFyZSB0d28gcHJvbmdlZDoNCj4gPiAtCSAqICAgLUZvciBzdHJlYW1pbmcgZGF0YSwgZWxpZGVz IG5lZWQgZm9yIGNhY2hlIG1haW50ZW5hbmNlLCBzYXZpbmcNCj4gPiAtCSAqICAgIGN5Y2xlcyBp biBmbHVzaCBjb2RlLCBhbmQgYnVzIGJhbmR3aWR0aCBhcyBhbGwgdGhlIGxpbmVzIG9mIGENCj4g PiAtCSAqICAgIGJ1ZmZlciBuZWVkIHRvIGJlIGZsdXNoZWQgb3V0IHRvIG1lbW9yeQ0KPiA+IC0J ICogICAtRm9yIGNvaGVyZW50IGRhdGEsIFJlYWQvV3JpdGUgdG8gYnVmZmVycyB0ZXJtaW5hdGUg ZWFybHkgaW4gY2FjaGUNCj4gPiAtCSAqICAgKHZzLiBhbHdheXMgZ29pbmcgdG8gbWVtb3J5IC0g dGh1cyBhcmUgZmFzdGVyKQ0KPiA+IC0JICovDQo+ID4gLQlpZiAoKGlzX2lzYV9hcmN2MigpICYm IGlvY19lbmFibGUpIHx8DQo+ID4gLQkgICAgKGF0dHJzICYgRE1BX0FUVFJfTk9OX0NPTlNJU1RF TlQpKQ0KPiA+ICsJaWYgKGF0dHJzICYgRE1BX0FUVFJfTk9OX0NPTlNJU1RFTlQpDQo+ID4gIAkJ bmVlZF9jb2ggPSAwOw0KPiA+ICANCj4gPiAgCS8qDQo+ID4gQEAgLTk1LDggKzgzLDcgQEAgdm9p ZCBhcmNoX2RtYV9mcmVlKHN0cnVjdCBkZXZpY2UgKmRldiwgc2l6ZV90IHNpemUsIHZvaWQgKnZh ZGRyLA0KPiA+ICAJc3RydWN0IHBhZ2UgKnBhZ2UgPSB2aXJ0X3RvX3BhZ2UocGFkZHIpOw0KPiA+ ICAJaW50IGlzX25vbl9jb2ggPSAxOw0KPiA+ICANCj4gPiAtCWlzX25vbl9jb2ggPSAoYXR0cnMg JiBETUFfQVRUUl9OT05fQ09OU0lTVEVOVCkgfHwNCj4gPiAtCQkJKGlzX2lzYV9hcmN2MigpICYm IGlvY19lbmFibGUpOw0KPiA+ICsJaXNfbm9uX2NvaCA9IChhdHRycyAmIERNQV9BVFRSX05PTl9D T05TSVNURU5UKTsNCj4gPiAgDQo+ID4gIAlpZiAoUGFnZUhpZ2hNZW0ocGFnZSkgfHwgIWlzX25v bl9jb2gpDQo+ID4gIAkJaW91bm1hcCgodm9pZCBfX2ZvcmNlIF9faW9tZW0gKil2YWRkcik7DQo+ ID4gQEAgLTE4MiwzICsxNjksMjAgQEAgdm9pZCBhcmNoX3N5bmNfZG1hX2Zvcl9jcHUoc3RydWN0 IGRldmljZSAqZGV2LCBwaHlzX2FkZHJfdCBwYWRkciwNCj4gPiAgCQlicmVhazsNCj4gPiAgCX0N Cj4gPiAgfQ0KPiANCj4gSSB0aGluayB3ZSBoYXZlIHNvbWUgc2hlbmFuaWdhbnMgd2l0aCBAaW9j X2VuYWJsZSBub3cuDQo+IERvIG5vdGUgdGhhdCBpdCB3YXMgbW9yZSBvZiBhIGRlYnVnIGhhY2sg dXNpbmcgd2hlbiB0aGUgaHcgZmVhdHVyZSB3YXMgaW50cm9kdWNlZA0KPiB0byBiZSBhYmxlIHRv IHJ1biBzYW1lIGtlcm5lbCBvbiB2YXJpb3VzIEZQR0EgYml0ZmlsZXMgYnV0IGp1c3QgZmxpY2tp bmcgYSBnbG9iYWwNCj4gdmFyaWFibGUgdmlhIHRoZSBkZWJ1Z2dlci4NCj4gDQo+IFNvIHBlciBj b2RlIGJlbG93LCBpZiBAaW9jX2VuYWJsZSBpcyBOT1Qgc2V0LCB3ZSBzdGlsbCB1c2Ugc29mdHdh cmUgYXNzaXN0ZWQgY2FjaGUNCj4gbWFpbnRlbmFuY2UsIGJ1dCBkbWFfe2FsbG9jLGZyZWV9IGRv bid0IGRvIHVzZSB0aGF0IHZhcmlhYmxlLiBIYXZlIHlvdSB0cmllZA0KPiB0ZXN0aW5nIHRoZSBj b21iaW5hdGlvbiB3aGVuIEBpb2NfZW5hYmxlIGlzIHNldCB0byAwIGJlZm9yZSBib290ID8gQW5k IGlzIHRoYXQgd29ya3MgPw0KDQpZZXAsIEkgdGVzdGVkIHRoYXQuDQpBbmQgaXQgd29ya3MgZmlu ZSB3aXRoIGJvdGggQGlvY19lbmFibGUgPT0gMCBhbmQgQGlvY19lbmFibGUgPT0gMQ0KTm90ZSB0 aGF0IHdlIGNoZWNrIHRoaXMgdmFyaWFibGUgaW4gYXJjaF9zZXR1cF9kbWFfb3BzKCkgZnVuY3Rp b24gbm93Lg0KDQpTbyB0aGlzIGFyY2hfZG1hX3thbGxvYyxmcmVlfSBhcmUgdXNlZCBPTkxZIGlu IGNhc2Ugb2Ygc29mdHdhcmUgYXNzaXN0ZWQgY2FjaGUgbWFpbnRlbmFuY2UuDQpUaGF0J3Mgd2h5 IHdlIGhhZCB0byBkbyBNTVUgbWFwcGluZyB0byBlbmZvcmNlIG5vbi1jYWNoYWJpbGl0eSByZWdh cmRsZXNzIG9mIEBpb2NfZW5hYmxlLg0KDQoNClByZXZpb3VzbHkgW2JlZm9yZSB0aGlzIHBhdGNo XSB3ZSB1c2VkIHRoaXMgb3BzIGZvciBib3RoIEhXL1NXIGFzc2lzdGVkIGNhY2hlIG1haW50ZW5h bmNlDQp0aGF0J3Mgd2h5IHdlIGNoZWNrZWQgQGlvY19lbmFibGUgaW4gYXJjaF9kbWFfe2FsbG9j LGZyZWV9LiAoaW4gY2FzZSBvZiBIVyBhc3Npc3RlZCBjYWNoZQ0KbWFpbnRlbmFuY2Ugd2Ugb25s eSBhbGxvY2F0ZSBtZW1vcnksIGFuZCBpbiBjYXNlIG9mIFNXIGFzc2lzdGVkIGNhY2hlIG1haW50 ZW5hbmNlIHdlIA0KYWxsb2NhdGUgbWVtb3J5IGFuZCBkbyBNTVUgbWFwcGluZyB0byBlbmZvcmNl IG5vbi1jYWNoYWJpbGl0eSkNCg0KPiA+ICsNCj4gPiArdm9pZCBhcmNoX3NldHVwX2RtYV9vcHMo c3RydWN0IGRldmljZSAqZGV2LCB1NjQgZG1hX2Jhc2UsIHU2NCBzaXplLA0KPiA+ICsJCQljb25z dCBzdHJ1Y3QgaW9tbXVfb3BzICppb21tdSwgYm9vbCBjb2hlcmVudCkNCj4gPiArew0KPiA+ICsJ LyoNCj4gPiArCSAqIElPQyBoYXJkd2FyZSBzbm9vcHMgYWxsIERNQSB0cmFmZmljIGtlZXBpbmcg dGhlIGNhY2hlcyBjb25zaXN0ZW50DQo+ID4gKwkgKiB3aXRoIG1lbW9yeSAtIGVsaWRpbmcgbmVl ZCBmb3IgYW55IGV4cGxpY2l0IGNhY2hlIG1haW50ZW5hbmNlIG9mDQo+ID4gKwkgKiBETUEgYnVm ZmVycyAtIHNvIHdlIGNhbiB1c2UgZG1hX2RpcmVjdCBjYWNoZSBvcHMuDQo+ID4gKwkgKi8NCj4g PiArCWlmIChpc19pc2FfYXJjdjIoKSAmJiBpb2NfZW5hYmxlICYmIGNvaGVyZW50KSB7DQo+ID4g KwkJc2V0X2RtYV9vcHMoZGV2LCAmZG1hX2RpcmVjdF9vcHMpOw0KPiA+ICsJCWRldl9pbmZvKGRl diwgInVzZSBkbWFfZGlyZWN0X29wcyBjYWNoZSBvcHNcbiIpOw0KPiA+ICsJfSBlbHNlIHsNCj4g PiArCQlzZXRfZG1hX29wcyhkZXYsICZkbWFfbm9uY29oZXJlbnRfb3BzKTsNCj4gPiArCQlkZXZf aW5mbyhkZXYsICJ1c2UgZG1hX25vbmNvaGVyZW50X29wcyBjYWNoZSBvcHNcbiIpOw0KPiA+ICsJ fQ0KPiA+ICt9DQo+IA0KPiANCi0tIA0KIEV1Z2VuaXkgUGFsdHNldg== From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eugeniy.Paltsev@synopsys.com (Eugeniy Paltsev) Date: Mon, 13 Aug 2018 17:08:10 +0000 Subject: [PATCH v2 2/4] ARC: allow to use IOC and non-IOC DMA devices simultaneously In-Reply-To: References: <20180730162636.3556-1-Eugeniy.Paltsev@synopsys.com> <20180730162636.3556-3-Eugeniy.Paltsev@synopsys.com> List-ID: Message-ID: <1534180089.3962.68.camel@synopsys.com> To: linux-snps-arc@lists.infradead.org On Mon, 2018-08-13@16:24 +0000, Vineet Gupta wrote: > On 07/30/2018 09:26 AM, Eugeniy Paltsev wrote: > > @@ -1263,11 +1254,7 @@ void __init arc_cache_init_master(void) > > if (is_isa_arcv2() && ioc_enable) > > arc_ioc_setup(); > > > > - if (is_isa_arcv2() && ioc_enable) { > > - __dma_cache_wback_inv = __dma_cache_wback_inv_ioc; > > - __dma_cache_inv = __dma_cache_inv_ioc; > > - __dma_cache_wback = __dma_cache_wback_ioc; > > - } else if (is_isa_arcv2() && l2_line_sz && slc_enable) { > > + if (is_isa_arcv2() && l2_line_sz && slc_enable) { > > __dma_cache_wback_inv = __dma_cache_wback_inv_slc; > > __dma_cache_inv = __dma_cache_inv_slc; > > __dma_cache_wback = __dma_cache_wback_slc; > > diff --git a/arch/arc/mm/dma.c b/arch/arc/mm/dma.c > > index cefb776a99ff..4d1466905e48 100644 > > --- a/arch/arc/mm/dma.c > > +++ b/arch/arc/mm/dma.c > > @@ -33,19 +33,7 @@ void *arch_dma_alloc(struct device *dev, size_t size, dma_addr_t *dma_handle, > > if (!page) > > return NULL; > > > > - /* > > - * IOC relies on all data (even coherent DMA data) being in cache > > - * Thus allocate normal cached memory > > - * > > - * The gains with IOC are two pronged: > > - * -For streaming data, elides need for cache maintenance, saving > > - * cycles in flush code, and bus bandwidth as all the lines of a > > - * buffer need to be flushed out to memory > > - * -For coherent data, Read/Write to buffers terminate early in cache > > - * (vs. always going to memory - thus are faster) > > - */ > > - if ((is_isa_arcv2() && ioc_enable) || > > - (attrs & DMA_ATTR_NON_CONSISTENT)) > > + if (attrs & DMA_ATTR_NON_CONSISTENT) > > need_coh = 0; > > > > /* > > @@ -95,8 +83,7 @@ void arch_dma_free(struct device *dev, size_t size, void *vaddr, > > struct page *page = virt_to_page(paddr); > > int is_non_coh = 1; > > > > - is_non_coh = (attrs & DMA_ATTR_NON_CONSISTENT) || > > - (is_isa_arcv2() && ioc_enable); > > + is_non_coh = (attrs & DMA_ATTR_NON_CONSISTENT); > > > > if (PageHighMem(page) || !is_non_coh) > > iounmap((void __force __iomem *)vaddr); > > @@ -182,3 +169,20 @@ void arch_sync_dma_for_cpu(struct device *dev, phys_addr_t paddr, > > break; > > } > > } > > I think we have some shenanigans with @ioc_enable now. > Do note that it was more of a debug hack using when the hw feature was introduced > to be able to run same kernel on various FPGA bitfiles but just flicking a global > variable via the debugger. > > So per code below, if @ioc_enable is NOT set, we still use software assisted cache > maintenance, but dma_{alloc,free} don't do use that variable. Have you tried > testing the combination when @ioc_enable is set to 0 before boot ? And is that works ? Yep, I tested that. And it works fine with both @ioc_enable == 0 and @ioc_enable == 1 Note that we check this variable in arch_setup_dma_ops() function now. So this arch_dma_{alloc,free} are used ONLY in case of software assisted cache maintenance. That's why we had to do MMU mapping to enforce non-cachability regardless of @ioc_enable. Previously [before this patch] we used this ops for both HW/SW assisted cache maintenance that's why we checked @ioc_enable in arch_dma_{alloc,free}. (in case of HW assisted cache maintenance we only allocate memory, and in case of SW assisted cache maintenance we allocate memory and do MMU mapping to enforce non-cachability) > > + > > +void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size, > > + const struct iommu_ops *iommu, bool coherent) > > +{ > > + /* > > + * IOC hardware snoops all DMA traffic keeping the caches consistent > > + * with memory - eliding need for any explicit cache maintenance of > > + * DMA buffers - so we can use dma_direct cache ops. > > + */ > > + if (is_isa_arcv2() && ioc_enable && coherent) { > > + set_dma_ops(dev, &dma_direct_ops); > > + dev_info(dev, "use dma_direct_ops cache ops\n"); > > + } else { > > + set_dma_ops(dev, &dma_noncoherent_ops); > > + dev_info(dev, "use dma_noncoherent_ops cache ops\n"); > > + } > > +} > > -- Eugeniy Paltsev