Re: [PATCH v9 15/27] cxl: define a driver interface for HPA free space enumeration

Alejandro Lucero Palau
2025-01-03T07:55:14Z

On 1/2/25 15:10, Jonathan Cameron wrote:
> On Mon, 30 Dec 2024 21:44:33 +0000
> <alejandro.lucero-palau@amd.com> wrote:
>
>> From: Alejandro Lucero <alucerop@amd.com>
>>
>> CXL region creation involves allocating capacity from device DPA
>> (device-physical-address space) and assigning it to decode a given HPA
>> (host-physical-address space). Before determining how much DPA to
>> allocate the amount of available HPA must be determined. Also, not all
>> HPA is created equal, some specifically targets RAM, some target PMEM,
>> some is prepared for device-memory flows like HDM-D and HDM-DB, and some
>> is host-only (HDM-H).
>>
>> Wrap all of those concerns into an API that retrieves a root decoder
>> (platform CXL window) that fits the specified constraints and the
>> capacity available for a new region.
>>
>> Based on https://lore.kernel.org/linux-cxl/168592159290.1948938.13522227102445462976.stgit@dwillia2-xfh.jf.intel.com/
>>
>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
>> Co-developed-by: Dan Williams <dan.j.williams@intel.com>
> A couple of trivial things inline.
>
> With those tidied up.
>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>
>> ---
>> drivers/cxl/core/region.c | 155 ++++++++++++++++++++++++++++++++++++++
>> drivers/cxl/cxl.h | 3 +
>> include/cxl/cxl.h | 8 ++
>> 3 files changed, 166 insertions(+)
>>
>> diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
>> index 967132b49832..239fe49bf6a6 100644
>> --- a/drivers/cxl/core/region.c
>> +++ b/drivers/cxl/core/region.c
>> @@ -687,6 +687,161 @@ static int free_hpa(struct cxl_region *cxlr)
>> return 0;
>> }
>>
>> +struct cxlrd_max_context {
>> + struct device *host_bridge;
>> + unsigned long flags;
>> + resource_size_t max_hpa;
>> + struct cxl_root_decoder *cxlrd;
>> +};
>> +
>> +static int find_max_hpa(struct device *dev, void *data)
>> +{
>> + struct cxlrd_max_context *ctx = data;
>> + struct cxl_switch_decoder *cxlsd;
>> + struct cxl_root_decoder *cxlrd;
>> + struct resource *res, *prev;
>> + struct cxl_decoder *cxld;
>> + resource_size_t max;
>> +
>> + if (!is_root_decoder(dev))
>> + return 0;
>> +
>> + cxlrd = to_cxl_root_decoder(dev);
>> + cxlsd = &cxlrd->cxlsd;
>> + cxld = &cxlsd->cxld;
>> + if ((cxld->flags & ctx->flags) != ctx->flags) {
>> + dev_dbg(dev, "%s, flags not matching: %08lx vs %08lx\n",
>> + __func__, cxld->flags, ctx->flags);
> Another __func__ to drop.
> Check for other cases where dynamic debug applies.
>

I'll do.

>
>> +
>> +/**
>> + * cxl_get_hpa_freespace - find a root decoder with free capacity per constraints
>> + * @cxlmd: the CXL memory device with an endpoint that is mapped by the returned
>> + * decoder
>> + * @flags: CXL_DECODER_F flags for selecting RAM vs PMEM, and HDM-H vs HDM-D[B]
>> + * @max_avail_contig: output parameter of max contiguous bytes available in the
>> + * returned decoder
>> + *
>> + * The return tuple of a 'struct cxl_root_decoder' and 'bytes available given
>> + * in (@max_avail_contig))' is a point in time snapshot. If by the time the
>> + * caller goes to use this root decoder's capacity the capacity is reduced then
>> + * caller needs to loop and retry.
>> + *
>> + * The returned root decoder has an elevated reference count that needs to be
>> + * put with put_device(cxlrd_dev(cxlrd)). Locking context is with
> put_device(CXLRD_DEV(cxlrd))

Right.

This helps to see the reference to cxl_acquire/release_endpoint needs to
be dropped.

Thanks!

>> + * cxl_{acquire,release}_endpoint(), that ensures removal of the root decoder
>> + * does not race.
>> + */
>> +struct cxl_root_decoder *cxl_get_hpa_freespace(struct cxl_memdev *cxlmd,
>> + unsigned long flags,
>> + resource_size_t *max_avail_contig)
>> +{
>> + struct cxl_port *endpoint = cxlmd->endpoint;
>> + struct cxlrd_max_context ctx = {
>> + .host_bridge = endpoint->host_bridge,
>> + .flags = flags,
>> + };
>> + struct cxl_port *root_port;
>> + struct cxl_root *root __free(put_cxl_root) = find_cxl_root(endpoint);
>> +
>> + if (!is_cxl_endpoint(endpoint)) {
>> + dev_dbg(&endpoint->dev, "hpa requestor is not an endpoint\n");
>> + return ERR_PTR(-EINVAL);
>> + }
>> +
>> + if (!root) {
>> + dev_dbg(&endpoint->dev, "endpoint can not be related to a root port\n");
>> + return ERR_PTR(-ENXIO);
>> + }
>> +
>> + root_port = &root->port;
>> + down_read(&cxl_region_rwsem);
>> + device_for_each_child(&root_port->dev, &ctx, find_max_hpa);
>> + up_read(&cxl_region_rwsem);
>> +
>> + if (!ctx.cxlrd)
>> + return ERR_PTR(-ENOMEM);
>> +
>> + *max_avail_contig = ctx.max_hpa;
>> + return ctx.cxlrd;
>> +}
>> +EXPORT_SYMBOL_NS_GPL(cxl_get_hpa_freespace, "CXL");
>> +