Linux DMA address

Linux DMA address

[Klaus Wacker]

Hello,
in the context of ib memory management on Linux we are going to use
ib_get_dma_mr() in conjunction with ib_dma_map_single(). The
ib_dma_map_single is described as "Map a kernel virtual address to DMA
address", i have a question on the returned value "DMA address". I learned
somewhere that DMA address is not necessarily a 64-bit real address.

When we call ib_dma_map_single() with same buffer-address, size and
dma_data_direction, but with different ib_devs. Would that generate the
same Linux DMA address on all ib devices? Sure we will have controls in
place that not both devices write into the buffer simultaneously. So whats
the nature of the returned DMA address of ib_dma_map_single and can we
assume that it is the same value on different devices on one system?
Thanks for your time spending on this topic.

Kind regards
Klaus Wacker
IBM Systems &Technology Group, Systems Software Development
SW Linux on System z Dev & Service
Phone: 	+49-7031-16-3779 	 IBM Deutschland 	E-Mail:
kdwacker-tA70FqPdS9bQT0dZR+AlfA@public.gmane.org 	 Schoenaicher Str. 220		 71032 Boeblingen,
Germany
IBM Deutschland Research & Development GmbH / Vorsitzende des
Aufsichtsrats: Martina Koederitz
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux DMA address

[Or Gerlitz]

On 7/2/2012 2:44 PM, Klaus Wacker wrote:
> [...] So whats the nature of the returned DMA address of ib_dma_map_single and can we
> assume that it is the same value on different devices on one system?

Currently, over all IB drivers/devices except for ipath and qib, 
ib_dma_map_single is just plain dma_map_single, so your questions 
relates directly to the Linux DMA mapping API (and BTW - the answer is 
YES, in the general case you must use a DMA address you were provided 
when calling dma_map_single with device being X only for transactions 
carried over X). For those devices that hack ib_dma_map_single, it might 
be possible to use address provided from device X over device Y, but 
this is a hack, and I would strongly recommend not to have it in you code.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux DMA address

[Klaus Wacker]

Hi,
thanks for your quick answer.
Let me stress this once again...
1.) Is it possible/allowed to call ib_dma_map_single() on Linux, same
buffer address, size and dma_data direction, for two different ib devices?
For the data transmission we will have controls in place that not both
devices write at the same time into that buffer.
2.) If this is possible, will the returned dma_addresses be the same? Any
dependencies on the adapter model?

Any thoughts on this? Thanks.
Kind regards
Klaus Wacker
IBM Systems &Technology Group, Systems Software Development
SW Linux on System z Dev & Service
Phone:  +49-7031-16-3779   IBM Deutschland  E-Mail:  kdwacker-tA70FqPdS9Zhl2p70BpVqQ@public.gmane.orgm
Schoenaicher Str. 220   71032 Boeblingen   Germany
IBM Deutschland Research & Development GmbH / Vorsitzende des
Aufsichtsrats: Martina Koederitz
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294


|------------>
| From:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>                                                                                                                |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To:        |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |Klaus Wacker/Germany/IBM@IBMDE,                                                                                                                   |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Cc:        |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |<linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>                                                                                                                      |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |02/07/2012 15:54                                                                                                                                  |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject:   |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |Re: Linux DMA address                                                                                                                             |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|





On 7/2/2012 2:44 PM, Klaus Wacker wrote:
> [...] So whats the nature of the returned DMA address of
ib_dma_map_single and can we
> assume that it is the same value on different devices on one system?

Currently, over all IB drivers/devices except for ipath and qib,
ib_dma_map_single is just plain dma_map_single, so your questions
relates directly to the Linux DMA mapping API (and BTW - the answer is
YES, in the general case you must use a DMA address you were provided
when calling dma_map_single with device being X only for transactions
carried over X). For those devices that hack ib_dma_map_single, it might
be possible to use address provided from device X over device Y, but
this is a hack, and I would strongly recommend not to have it in you code.

Or.




--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux DMA address

[Or Gerlitz]

> Let me stress this once again...
> 1.) Is it possible/allowed to call ib_dma_map_single() on Linux, same
> buffer address, size and dma_data direction, for two different ib devices?
> For the data transmission we will have controls in place that not both
> devices write at the same time into that buffer.
> 2.) If this is possible, will the returned dma_addresses be the same? Any
> dependencies on the adapter model?
> Any thoughts on this? Thanks.

Here's the question for you - can you strip the ib_ part from the
mentioned APIs and answer the question? e.g w.r.t your non IB HW of
choice (still under Linux)? if yes, the same answer applies with the
ib_ part, if not, I can try to help, but please  take a look 1st on
the DMA mapping documents under the kernel doc dir.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux DMA address

[Roland Dreier]

On Tue, Jul 3, 2012 at 6:37 AM, Klaus Wacker <Klaus.Wacker-tA70FqPdS9bQT0dZR+AlfA@public.gmane.org> wrote:
> 1.) Is it possible/allowed to call ib_dma_map_single() on Linux, same
> buffer address, size and dma_data direction, for two different ib devices?

Yes, this should be no problem.

> 2.) If this is possible, will the returned dma_addresses be the same? Any
> dependencies on the adapter model?

There is no guarantee about what the addresses will be.  For example
if the adapters are behind IOMMUs (eg Intel VT-d) then it is quite likely
the dma addresses will be different for each ib device.

As Or has said, ib_dma_xxx functions are pretty thin wrappers around
the normal dma api -- they're just there to handle ib devices like qib/ipath
that emulate dma in software.  So if you understand the dma api then
the same answer should apply to the ib_dma_ functions.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html