Re: Translation registers

[Dave Jiang <dave.jiang@intel.com>]

On 08/21/2017 02:59 PM, Gary R Hook wrote:
> Stupid question time.
> 
> What is the purpose of the call to dma_alloc_coherent() in ntb_perf and
> ntb_tool? Is is supposed to acquire a chunk of DMA memory that is mapped
> into the device bar? A call to ntb_mw_set_trans() ensues, but I'm not
> seeing what is supposed to happen in that, either. The translation
> registers are accessed, yes, but why?
> 
> I'm seeing the memory window this way: On my primary I get the MW
> address through the BAR, it's a certain size, and I can write into it.
> The data should appear in the corresponding memory window of the
> secondary. An application could be notified via a doorbell event to go
> read the data.
> 
> Or am I just completely confused?
> 
> What I'm trying to get down to is how, during initialization of the
> device driver and tool module, I get a virtual address that I can use
> for memcpy operations. The physical address (for DMA ops) is easy; I've
> got that. What's not clear is how (on Intel hardware) is how
> ioremap_wc()/et. al. are used to get a virtual reference to a physical
> address.
> 
> Searching has as yet revealed nothing specific on this detail. Of
> course, I could be making this more complex than it needs to be...
> 
> 
> In summary: how is the Intel NTB device getting mapped into the virtual
> address space, via dma_alloc_coherent, ioremap_wc, and ntb_mw_set_trans?
> 
> Much thanks to anyone that has illuminating comments.
> 
So in the case of ntb_transport or ntb_perf, the dma_alloc_coherent()
provides a chunk of DMAable memory to back the NTB BAR that the other
node will be accessing.

So let's use the terminology of node A and node B in a B2B (back to
back) configuration. On node B, your NTB memory window BAR has the
address location of 0x383fffe00000, which is set by BIOS. You allocate
via dma_alloc_coherent() on node A and get the physical address of
0x87f000000 for the memory. You would write this bus address into the
SBARXLAT (secondary BAR translate) register. The virtual address for
that piece of memory is provided to you as the return value when you
called dma_alloc_coherent(), lets use 0x20000000 for simplicity sake.

On node B, ioremap_*() is called in order to obtain a virtual address so
you can access the NTB memory window BAR via CPU. This is basically
standard PCI BAR mapping. You would get a virtual address returned that
maps to 0x383fffe00000. Lets for simplicity sake call it 0x10000000 for
that virtual address. So when you attempt to write to the virtual
address of 0x10001000, it would be 0x383fffe01000 accessed on the PCI
BAR on node B. It goes through NTB translations and would end up in the
memory address of 0x20001000 virtually and 0x383fffe01000 physical wise
on node A.

I hope that helps?

The doorbells are completely separate from the memory window access. The
software can choose to ring the doorbell after it has written the data.
The writing of the data does not trigger the doorbell.