Uncached buffers from CMA DMA heap on some Arm devices?

[Milan Zamazal <mzamazal@redhat.com>]

Hello,

in the libcamera project, we experience a major performance problem related to
DMA buffers while working on camera image processing using CPU.  This happens
only with some Arm boards, we have observed it on Debix Model A (NXP i.MX 8M
Plus) and PinePhone.  We use /dev/dma_heap/linux,cma (or reserved) DMA buffer
heap on Arm.

Reading V4L2 camera data from buffers is very slow.  When we memcpy the data
from the buffer to a malloc'ed memory before working with it (reading each byte
multiple times, without any big non-sequential jumps across the data), we get
more than 10 times speed up.  It looks like the input buffer is uncached.

We experience slow down also when writing to output buffers.  It doesn't seem to
matter whether we write to the output byte-by-byte or memcpy larger chunks.

We are having trouble to understand what's the problem with the buffers on some
hardware and what we can realistically do about it.  Could you please help us
clarify this?  Is it possible to force the DMA buffer CMA heap to be cached?
Or is there anything else we can do or try?

Thank you,
Milan