[LSF/MM/BPF TOPIC] [LSF/MM/BPF ATTEND] : Two stage IOMMU DMA mapping operations

Chaitanya Kulkarni
2024-02-27T08:17:47Z

Hi,

* Problem Statement :-
-------------------------------------------------------------------------
The existing IOMMU DMA mapping operation is performed in two steps at the
same time (one-shot):
1. Allocates IOVA space.
2. Actually maps DMA pages to that space.
For example, map scatter-gather list:
dma_map_sg_attrs()
__dma_map_sg_attrs
ops->map_sg()
iommu_dma_map_sg()
Calculate length of IOVA space that  is needed

/* ####### step one allocate IOVA space ####### */
iommu_dma_alloc_iova()

/* ####### step two actually map DMA Pages ####### */
iommu_map_sg()
for each entry in sg list()
__iommu_map()
iommu_domain_ops->map_pages()

This one-shot operation works perfectly for non-complex scenarios where
callers use the existing DMA API in the control path when they setup
hardware.

However, in more complex scenarios, when DMA mapping is needed in the
data path and especially when some sort of specific intermediary
datatype is involved (sg list), this one-shot approach:

1. Forces developers to introduce new DMA APIs for specific datatype,
e.g., Existing scatter-gather mapping functions in dma mapping
existing subsystems :-

dma_map_sgtable()
__dma_map_sg_attrs()
dma_unmap_sg_attrs()
blk_rq_map_sg()
__blk_rq_map_sg()
__blk_bvec_map_sg()
__blk_bios_map_sg()
blk_bvec_map_sg()

OR

Latest Chuck's RFC series [1] aims to incorporate biovec-related
DMA mapping (which expands bio_vec with DMA addresses). Probably,
struct folio will also require it.

2. Creates dependencies on a data type, forcing certain intermediary
data type allocation/de-allocation and page-to-data-type mapping
and unmapping in the fast path (submission or completion).

* Proposed approach and discussion points :-
-------------------------------------------------------------------------

Instead of teaching DMA APIs to know about specific datatypes & creating
a dependency on it, that may add performance overhead with mapping and
allocation, we propose to separate the existing DMA mapping routine into
two steps where:

Step 1 : Provide an option to API users (subsystems) to perform all
calculations internally in-advance.
Step 2 : Map pages when they are needed.

These advanced DMA mapping APIs are needed to calculate the IOVA size to
allocate as one chunk and a combination of offset calculations to know
which part of IOVA to be mapped to which page.

The new API will also allow us to remove the dependency on the sg list as
discussed previously in [2].

The main advantages of this approach as it is seen in upcoming RFC
series are:

1. Simplified & increased performance in page fault handling for
On-Demand-Paging (ODP) mode for RDMA.
2. Reduced memory footprint for VFIO PCI live migration code.
3. Reduced overhead of intermediary SG table manipulation in the fast
path for storage drivers where block layer requests are mapped onto
sg table and then sg table is mapped onto DMA :-
xxx_queue_rq()
allocate sg table
blk_rq_map_sg()
merge and maps bvecs to sg
dma_map_sgtable()
maps pages in sg to DMA.

In order to create a good platform for a concrete and meaningful
discussion at LSFMM 24, we plan to post an RFC within the next two weeks.

Required Attendees list :-

Christoph Hellwig
Jason Gunthorpe
Jens Axboe
Chuck Lever
David Howells
Keith Busch
Bart Van Assche
Damien Le Moal
Martin Petersen

-ck

[1]
https://lore.kernel.org/all/169772852492.5232.17148564580779995849.stgit@klimt.1015granger.net
[2] https://lore.kernel.org/linux-iommu/20200708065014.GA5694@lst.de/