Re: [PATCH RFC 00/12] dma: Enable dmem cgroup tracking

Simona Vetter
2025-04-03T08:27:24Z

On Mon, Mar 10, 2025 at 01:06:06PM +0100, Maxime Ripard wrote:
> Hi,
>
> Here's preliminary work to enable dmem tracking for heavy users of DMA
> allocations on behalf of userspace: v4l2, DRM, and dma-buf heaps.
>
> It's not really meant for inclusion at the moment, because I really
> don't like it that much, and would like to discuss solutions on how to
> make it nicer.
>
> In particular, the dma dmem region accessors don't feel that great to
> me. It duplicates the logic to select the proper accessor in
> dma_alloc_attrs(), and it looks fragile and potentially buggy to me.
>
> One solution I tried is to do the accounting in dma_alloc_attrs()
> directly, depending on a flag being set, similar to what __GFP_ACCOUNT
> is doing.
>
> It didn't work because dmem initialises a state pointer when charging an
> allocation to a region, and expects that state pointer to be passed back
> when uncharging. Since dma_alloc_attrs() returns a void pointer to the
> allocated buffer, we need to put that state into a higher-level
> structure, such as drm_gem_object, or dma_buf.
>
> Since we can't share the region selection logic, we need to get the
> region through some other mean. Another thing I consider was to return
> the region as part of the allocated buffer (through struct page or
> folio), but those are lost across the calls and dma_alloc_attrs() will
> only get a void pointer. So that's not doable without some heavy
> rework, if it's a good idea at all.
>
> So yeah, I went for the dumbest possible solution with the accessors,
> hoping you could suggest a much smarter idea :)

I've had a private chat with Maxime to get him up to speed on hopefully a
lot of the past discussions, but probably best I put my notes here too.
Somewhat unstructured list of challenges with trying to account all the
memory for gpu/isp/camera/whatever:

- At LPC in Dublin I think we've pretty much reached the conclusion that
normal struct page memory should be just accounted in memcg. Otherwise
you just get really nasty double-accounting chaos or issues where you
can exhaust reserves.

- We did not figure out what to do with mixed stuff like CMA, where we
probably want to account it both into memcg (because it's struct page)
but also separately into dmem (because the CMA region is a limited
resource and only using memcg will not help us manage it).

- There's the entire chaos of carve-out vs CMA and how userspace can
figure out how to set reasonable limits automatically. Maxime brought
the issue that limits need to be adjusted if carve-out/CMA/shmem aren't
accounted the same, which I think is a valid concern. But due to the
above conclusion around memcg accounting I think that's unavoidable, so
we need some means for userspace to autoconfigure reasonable limits.
Then that autoconfig can be done on each boot, and kernel (or dt or
whatever) changes between these three allocators don't matter anymore.

- Autoconfiguration challenges also exist for split display/render SoC. It
gets even more fun if you also throw in camera and media codecs, and
even more fun if you have multiple CMA regions.

- Discrete gpu also has a very fun autoconfiguration issue because you
have dmem limits for vram, and memcg limits for system memory. Vram
might be swapped out to system memory, so naively you might want to
assume that you need higher memcg limits than dmem limits. But there's
systems with more dmem and smem (because the cpu with its memory is
essentially just the co-processor that orchestrates the real compute
machine, which is all gpus).

- We need a charge transfer, least for Android since there all memory is
allocated through binder. TJ Mercier did some patches:

https://lore.kernel.org/dri-devel/20230123191728.2928839-3-tjmercier@google.com/

Ofc with dmem this would need to work for both dmem and memcg charges,
since with CMA and discrete gpu we'll have bo that are tracked in both.

- Hard limits for shmem/ttm drivers need a memcg-aware shrinker. TTM
doesn't even have a shrinker yet, but with xe we now have a
helper-library approach to enabling shrinking for TTM drivers.
memcg-aware shrinking will be a large step up in complexity on top (and
probably a good reason to switch over to the common shrinker lru instead
of hand-rolling).

See the various attempts at ttm shrinkers by Christian König and Thomas
Hellstrom over the past years on dri-devel.

This also means that most likely cgroup limit enforcement for ttm based
drivers will be per-driver or at least very uneven.

- Hard limits for dmem vram means ttm eviction needs to be able to account
the evicted bo against the right memcg. Because this can happen in
random other threads (cs ioctl of another process, kernel threads)
accounting this correctly is going to be "fun". Plus I haven't thought
through interactions with memcg-aware shrinkers, which might cause some
really fundamental issues.

- We also ideally need pin account, but I don't think we have any
consensus on how to do that for memcg memory. Thus far it's all
functionality-specific limits (e.g. mlock, rdma has its own for
long-term pinned memory), not sure it makes sense to push for a unified
tracking in memcg here?

For dmem I think it's pretty easy, but there the question is how to
differentiate between dmem that's always pinned (cma, I don't think
anyone bothered with a shrinker for cma memory, vc4 maybe?) and dmem
that generally has a shrinker and really wants a separate pin limit
(vram/ttm drivers).

- Unfortunately on top of the sometimes very high individual complexity
these issues also all interact. Which means that we won't be able to
roll this out in one go, and we need to cope with very uneven
enforcement. I think trying to allow userspace to cope with changing
cgroup support through autoconfiguration is the most feasible way out of
this challenge.

tldr; cgroup for device memory is a really complex mess

Cheers, Sima

> Thanks,
> Maxime
>
> Signed-off-by: Maxime Ripard <mripard@kernel.org>
> ---
> Maxime Ripard (12):
> cma: Register dmem region for each cma region
> cma: Provide accessor to cma dmem region
> dma: coherent: Register dmem region for each coherent region
> dma: coherent: Provide accessor to dmem region
> dma: contiguous: Provide accessor to dmem region
> dma: direct: Provide accessor to dmem region
> dma: Create default dmem region for DMA allocations
> dma: Provide accessor to dmem region
> dma-buf: Clear cgroup accounting on release
> dma-buf: cma: Account for allocations in dmem cgroup
> drm/gem: Add cgroup memory accounting
> media: videobuf2: Track buffer allocations through the dmem cgroup
>
> drivers/dma-buf/dma-buf.c | 7 ++++
> drivers/dma-buf/heaps/cma_heap.c | 18 ++++++++--
> drivers/gpu/drm/drm_gem.c | 5 +++
> drivers/gpu/drm/drm_gem_dma_helper.c | 6 ++++
> .../media/common/videobuf2/videobuf2-dma-contig.c | 19 +++++++++++
> include/drm/drm_device.h | 1 +
> include/drm/drm_gem.h | 2 ++
> include/linux/cma.h | 9 +++++
> include/linux/dma-buf.h | 5 +++
> include/linux/dma-direct.h | 2 ++
> include/linux/dma-map-ops.h | 32 ++++++++++++++++++
> include/linux/dma-mapping.h | 11 ++++++
> kernel/dma/coherent.c | 26 +++++++++++++++
> kernel/dma/direct.c | 8 +++++
> kernel/dma/mapping.c | 39 ++++++++++++++++++++++
> mm/cma.c | 21 +++++++++++-
> mm/cma.h | 3 ++
> 17 files changed, 211 insertions(+), 3 deletions(-)
> ---
> base-commit: 55a2aa61ba59c138bd956afe0376ec412a7004cf
> change-id: 20250307-dmem-cgroups-73febced0989
>
> Best regards,
> --
> Maxime Ripard <mripard@kernel.org>
>

--
Simona Vetter
Software Engineer, Intel Corporation