Re: [RFC PATCH v4 5/6] drivers/migrate_offload: add DMA batch copy driver (dcbm)

["Garg, Shivank" <shivankg@amd.com>]

Hi Vinod,

Following your suggestion at the Kernel meetup in Bangalore (11 Apr 2026)
to check 0cae04373b ("dmaengine: remove DMA_MEMCPY_SG once again") and use
DMA_MEMCPY_SG / dmaengine_prep_dma_memcpy_sg() (I added a
device_prep_dma_memcpy_sg hook in drivers/dma/amd/ptdma/ptdma-dmaengine.c
for this experiment; not posted).
I ran an A/B comparison against the existing DCBM path that uses
dmaengine_prep_dma_memcpy() in a loop over mapped SGL segments.

I'm using the move_pages() workload to move 1 GB data per run. I do not see
significant performance difference, and results are broadly within each
other's noise band).

Throughput (GB/s, mean ± SD), ITERATIONS=10:

Page     nr_dma_chan=1              nr_dma_chan=4              nr_dma_chan=8              nr_dma_chan=16
order    dcbm          dcbm_sg      dcbm          dcbm_sg      dcbm          dcbm_sg      dcbm           dcbm_sg
------   -----------   ----------   -----------   ----------   -----------   ----------   ------------   ----------
0        2.33 ± 0.17   2.26 ± 0.19  3.24 ± 0.21   3.18 ± 0.23  3.29 ± 0.10   3.45 ± 0.10  3.29 ± 0.13    3.49 ± 0.22
4        2.77 ± 0.21   2.99 ± 0.18  6.26 ± 0.99   6.75 ± 0.12  8.01 ± 0.58   7.70 ± 0.64  8.22 ± 0.89    8.72 ± 0.87
8        4.57 ± 0.70   4.75 ± 0.83  10.64 ± 1.97  10.94 ± 3.52 10.30 ± 1.22  10.36 ± 1.24 11.27 ± 1.21   12.47 ± 1.66
9        12.71 ± 0.09  12.68 ± 0.08 27.13 ± 0.15  26.89 ± 0.27 46.50 ± 0.73  45.17 ± 2.46 67.25 ± 1.42   62.78 ± 8.24

Notes: order 0/4/8/9 = 4K / 64K / 1M / 2M folios
	dcbm     = per-segment dmaengine_prep_dma_memcpy
	dcbm_sg  = DMA_MEMCPY_SG / dmaengine_prep_dma_memcpy_sg

<snip>

> +
> +static int submit_dma_transfers(struct dma_work *work)
> +{
> +	struct scatterlist *sg_src, *sg_dst;
> +	struct dma_async_tx_descriptor *tx;
> +	unsigned long flags = DMA_CTRL_ACK;
> +	dma_cookie_t cookie;
> +	int i;
> +
> +	atomic_set(&work->pending, 1);
> +
> +	sg_src = work->src_sgt->sgl;
> +	sg_dst = work->dst_sgt->sgl;
> +	for_each_sgtable_dma_sg(work->src_sgt, sg_src, i) {
> +		if (i == work->src_sgt->nents - 1)
> +			flags |= DMA_PREP_INTERRUPT;
> +
> +		tx = dmaengine_prep_dma_memcpy(work->chan,
> +				sg_dma_address(sg_dst),
> +				sg_dma_address(sg_src),
> +				sg_dma_len(sg_src), flags);
> +		if (!tx) {
> +			atomic_set(&work->pending, 0);
> +			return -EIO;
> +		}
> +
> +		if (i == work->src_sgt->nents - 1) {
> +			tx->callback = dma_completion_callback;
> +			tx->callback_param = work;
> +		}
> +
> +		cookie = dmaengine_submit(tx);
> +		if (dma_submit_error(cookie)) {
> +			atomic_set(&work->pending, 0);
> +			return -EIO;
> +		}
> +		sg_dst = sg_next(sg_dst);
> +	}
> +	return 0;
> +}

static int submit_dma_transfers(struct dma_work *work)
{
        struct dma_async_tx_descriptor *tx;
        unsigned long flags = DMA_CTRL_ACK | DMA_PREP_INTERRUPT;
        dma_cookie_t cookie;

        tx = dmaengine_prep_dma_memcpy_sg(work->chan,
                        work->dst_sgt->sgl, work->dst_sgt->nents,
                        work->src_sgt->sgl, work->src_sgt->nents,
                        flags);
        if (!tx)
                return -EIO;

        atomic_set(&work->pending, 1);
        tx->callback = dma_completion_callback;
        tx->callback_param = work;

        cookie = dmaengine_submit(tx);
        if (dma_submit_error(cookie)) {
                atomic_set(&work->pending, 0);
                return -EIO;
        }
        return 0;
}

The memcpy_sg version does simplify submit_dma_transfers()
(one dmaengine_prep_dma_memcpy_sg + one dmaengine_submit vs a loop).

My current DCBM path issues dmaengine_prep_dma_memcpy()+dmaengine_submit()
per mapped SG segment and sets DMA_PREP_INTERRUPT + callback only
on the last one, so the IRQ/callback cost is already one per batch.

My understanding is switching to dmaengine_prep_dma_memcpy_sg() mainly
saves the per-segment prep/submit calls and hands the provider a single
multi-segment TX to program.

Please correct me if the benefit you had in mind is something stronger.
Thanks for the suggestion and for guidance.

Best Regards,
Shivank