Re: [PATCH rdma-next v2 0/3] cgroup/rdma: add MR memory size resource tracking

[Tao Cui <cui.tao@linux.dev>]

Hi Yanjun,

Thanks for the thoughtful questions.  FRWR is indeed a widely used
pattern, and the interaction with mr_mem deserves clarification.

> 1. Accounting Granularity: Does mr_mem charge the maximum capacity of
>    the FRWR object at its allocation time (ib_alloc_mr), or does it
>    dynamically track the actual mapped bytes during the fast-reg data
>    path?

In the current proposal, mr_mem is only charged for userspace MR
registrations that go through the uverbs layer (REG_MR, DM_MR,
DMABUF_MR, and the legacy ioctl path).  These are the paths where a
concrete byte length is known at registration time.

FRWR MRs allocated via ib_alloc_mr() are not charged for mr_mem.  The
actual registration footprint associated with an FRWR MR is not known
at allocation time: ib_alloc_mr() only specifies the maximum
scatter-gather capacity of the MR, while the mapped byte range may
change dynamically across successive ib_map_mr_sg() operations.

Supporting FRWR accounting would therefore require a separate
accounting model, since the registration footprint is established
dynamically rather than by a fixed length parameter supplied at MR
creation.  This is outside the scope of the current proposal.

> 2. Kernel-space vs Userspace: FRWR pools are frequently allocated by
>    kernel-space drivers (like NVMe-oF target/host). If these kernel
>    threads are not bound to a specific user cgroup, will their FRWR
>    allocations end up in the root cgroup, potentially bypassing the
>    per-tenant limits?

The RDMA cgroup's resource control is primarily designed for userspace
consumers.  Kernel-space consumers (NVMe-oF target, SRP initiator,
rtrs, iSER, etc.) allocate resources through kernel APIs
(ib_alloc_mr, ib_create_qp, etc.).  These resources do not currently
participate in RDMA cgroup accounting and therefore are not subject to
per-cgroup limits.

Kernel-space FRWR pools are typically managed by the administrator
rather than subject to per-tenant limits.

This behavior is consistent with the current RDMA cgroup model, which
tracks resources associated with userspace RDMA objects.  If accounting
were extended to kernel-allocated FRWR MRs, ownership semantics would
become an open question: simply charging against the current task or
the root cgroup may not accurately represent the tenant that ultimately
benefits from the resource.

> Don't you think it would be beneficial to explicitly document or
> consider the FRWR pattern in the design section, given its prevalence
> in real-world storage and networking workloads?

Agreed.  I will add a note to the cover letter and commit messages
clarifying that mr_mem currently covers only userspace MR registrations
with a known length, and that kernel-space FRWR pools are out of scope
for this initial proposal.  The semantic distinction between
userspace registration-length accounting and kernel-space FRWR
resource management is worth documenting explicitly.

Thanks,
Tao