From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga07.intel.com (mga07.intel.com. [134.134.136.100]) by gmr-mx.google.com with ESMTPS id v14si2306003pfd.7.2017.08.21.15.36.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Aug 2017 15:36:01 -0700 (PDT) Subject: Re: Translation registers References: <8f9eda70-cd72-b5b0-aa6b-30022cc96e7a@amd.com> From: Dave Jiang Message-ID: Date: Mon, 21 Aug 2017 15:36:00 -0700 MIME-Version: 1.0 In-Reply-To: <8f9eda70-cd72-b5b0-aa6b-30022cc96e7a@amd.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit To: Gary R Hook , linux-ntb@googlegroups.com List-ID: On 08/21/2017 02:59 PM, Gary R Hook wrote: > Stupid question time. > > What is the purpose of the call to dma_alloc_coherent() in ntb_perf and > ntb_tool? Is is supposed to acquire a chunk of DMA memory that is mapped > into the device bar? A call to ntb_mw_set_trans() ensues, but I'm not > seeing what is supposed to happen in that, either. The translation > registers are accessed, yes, but why? > > I'm seeing the memory window this way: On my primary I get the MW > address through the BAR, it's a certain size, and I can write into it. > The data should appear in the corresponding memory window of the > secondary. An application could be notified via a doorbell event to go > read the data. > > Or am I just completely confused? > > What I'm trying to get down to is how, during initialization of the > device driver and tool module, I get a virtual address that I can use > for memcpy operations. The physical address (for DMA ops) is easy; I've > got that. What's not clear is how (on Intel hardware) is how > ioremap_wc()/et. al. are used to get a virtual reference to a physical > address. > > Searching has as yet revealed nothing specific on this detail. Of > course, I could be making this more complex than it needs to be... > > > In summary: how is the Intel NTB device getting mapped into the virtual > address space, via dma_alloc_coherent, ioremap_wc, and ntb_mw_set_trans? > > Much thanks to anyone that has illuminating comments. > So in the case of ntb_transport or ntb_perf, the dma_alloc_coherent() provides a chunk of DMAable memory to back the NTB BAR that the other node will be accessing. So let's use the terminology of node A and node B in a B2B (back to back) configuration. On node B, your NTB memory window BAR has the address location of 0x383fffe00000, which is set by BIOS. You allocate via dma_alloc_coherent() on node A and get the physical address of 0x87f000000 for the memory. You would write this bus address into the SBARXLAT (secondary BAR translate) register. The virtual address for that piece of memory is provided to you as the return value when you called dma_alloc_coherent(), lets use 0x20000000 for simplicity sake. On node B, ioremap_*() is called in order to obtain a virtual address so you can access the NTB memory window BAR via CPU. This is basically standard PCI BAR mapping. You would get a virtual address returned that maps to 0x383fffe00000. Lets for simplicity sake call it 0x10000000 for that virtual address. So when you attempt to write to the virtual address of 0x10001000, it would be 0x383fffe01000 accessed on the PCI BAR on node B. It goes through NTB translations and would end up in the memory address of 0x20001000 virtually and 0x383fffe01000 physical wise on node A. I hope that helps? The doorbells are completely separate from the memory window access. The software can choose to ring the doorbell after it has written the data. The writing of the data does not trigger the doorbell.