Re: Translation registers

[Gary R Hook <gary.hook@amd.com>]

On 08/21/2017 05:36 PM, Dave Jiang wrote:
>
>
> On 08/21/2017 02:59 PM, Gary R Hook wrote:
>> Stupid question time.
>>
>> What is the purpose of the call to dma_alloc_coherent() in ntb_perf and
>> ntb_tool? Is is supposed to acquire a chunk of DMA memory that is mapped
>> into the device bar? A call to ntb_mw_set_trans() ensues, but I'm not
>> seeing what is supposed to happen in that, either. The translation
>> registers are accessed, yes, but why?
>>
>> I'm seeing the memory window this way: On my primary I get the MW
>> address through the BAR, it's a certain size, and I can write into it.
>> The data should appear in the corresponding memory window of the
>> secondary. An application could be notified via a doorbell event to go
>> read the data.
>>
>> Or am I just completely confused?
>>
>> What I'm trying to get down to is how, during initialization of the
>> device driver and tool module, I get a virtual address that I can use
>> for memcpy operations. The physical address (for DMA ops) is easy; I've
>> got that. What's not clear is how (on Intel hardware) is how
>> ioremap_wc()/et. al. are used to get a virtual reference to a physical
>> address.
>>
>> Searching has as yet revealed nothing specific on this detail. Of
>> course, I could be making this more complex than it needs to be...
>>
>>
>> In summary: how is the Intel NTB device getting mapped into the virtual
>> address space, via dma_alloc_coherent, ioremap_wc, and ntb_mw_set_trans?
>>
>> Much thanks to anyone that has illuminating comments.
>>
> So in the case of ntb_transport or ntb_perf, the dma_alloc_coherent()
> provides a chunk of DMAable memory to back the NTB BAR that the other
> node will be accessing.
> So let's use the terminology of node A and node B in a B2B (back to
> back) configuration. On node B, your NTB memory window BAR has the
> address location of 0x383fffe00000, which is set by BIOS. You allocate
> via dma_alloc_coherent() on node A and get the physical address of
> 0x87f000000 for the memory. You would write this bus address into the
> SBARXLAT (secondary BAR translate) register. The virtual address for
> that piece of memory is provided to you as the return value when you
> called dma_alloc_coherent(), lets use 0x20000000 for simplicity sake.
>
> On node B, ioremap_*() is called in order to obtain a virtual address so
> you can access the NTB memory window BAR via CPU. This is basically
> standard PCI BAR mapping. You would get a virtual address returned that
> maps to 0x383fffe00000. Lets for simplicity sake call it 0x10000000 for
> that virtual address. So when you attempt to write to the virtual
> address of 0x10001000, it would be 0x383fffe01000 accessed on the PCI
> BAR on node B. It goes through NTB translations and would end up in the
> memory address of 0x20001000 virtually and 0x383fffe01000 physical wise
> on node A.
>
> I hope that helps?

This is very nice, thank you. A and B are simpler; we just refer to them
as the primary and secondary.

In order to avoid embarrassing myself with a possible display of ignorance,
I'm going to have to study this in detail. But I will say that what you
describe above is not what is happening in my 4.12 kernel.

> The doorbells are completely separate from the memory window access. The
> software can choose to ring the doorbell after it has written the data.
> The writing of the data does not trigger the doorbell.

That was only intended as a usage strawman for a simple illustration. No
conflation there.

Thank you for your time. Greatly appreciated.