Re: [PATCH v9 15/27] cxl: define a driver interface for HPA free space enumeration

Dan Williams
2025-01-18T03:02:43Z

alejandro.lucero-palau@ wrote:
> From: Alejandro Lucero <alucerop@amd.com>
>
> CXL region creation involves allocating capacity from device DPA
> (device-physical-address space) and assigning it to decode a given HPA
> (host-physical-address space). Before determining how much DPA to
> allocate the amount of available HPA must be determined. Also, not all
> HPA is created equal, some specifically targets RAM, some target PMEM,
> some is prepared for device-memory flows like HDM-D and HDM-DB, and some
> is host-only (HDM-H).
>
> Wrap all of those concerns into an API that retrieves a root decoder
> (platform CXL window) that fits the specified constraints and the
> capacity available for a new region.
>
> Based on https://lore.kernel.org/linux-cxl/168592159290.1948938.13522227102445462976.stgit@dwillia2-xfh.jf.intel.com/

What needed changing such that you could not use the patch verbatim?
Then I can focus on that, although I am also critical of code I wrote
(like the DPA layout mess).

> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
> Co-developed-by: Dan Williams <dan.j.williams@intel.com>

Include Signed-off-by: whenever including Co-developed-by

> ---
> drivers/cxl/core/region.c | 155 ++++++++++++++++++++++++++++++++++++++
> drivers/cxl/cxl.h | 3 +
> include/cxl/cxl.h | 8 ++
> 3 files changed, 166 insertions(+)
>
> diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
> index 967132b49832..239fe49bf6a6 100644
> --- a/drivers/cxl/core/region.c
> +++ b/drivers/cxl/core/region.c
> @@ -687,6 +687,161 @@ static int free_hpa(struct cxl_region *cxlr)
> return 0;
> }
>
> +struct cxlrd_max_context {
> + struct device *host_bridge;
> + unsigned long flags;
> + resource_size_t max_hpa;
> + struct cxl_root_decoder *cxlrd;
> +};
> +
> +static int find_max_hpa(struct device *dev, void *data)
> +{
> + struct cxlrd_max_context *ctx = data;
> + struct cxl_switch_decoder *cxlsd;
> + struct cxl_root_decoder *cxlrd;
> + struct resource *res, *prev;
> + struct cxl_decoder *cxld;
> + resource_size_t max;
> +
> + if (!is_root_decoder(dev))
> + return 0;
> +
> + cxlrd = to_cxl_root_decoder(dev);
> + cxlsd = &cxlrd->cxlsd;
> + cxld = &cxlsd->cxld;
> + if ((cxld->flags & ctx->flags) != ctx->flags) {
> + dev_dbg(dev, "%s, flags not matching: %08lx vs %08lx\n",
> + __func__, cxld->flags, ctx->flags);
> + return 0;
> + }
> +
> + /*
> + * The CXL specs do not forbid an accelerator being part of an
> + * interleaved HPA range, but it is unlikely and because it simplifies
> + * the code, don´t allow it.
> + */
> + if (cxld->interleave_ways != 1) {
> + dev_dbg(dev, "interleave_ways not matching\n");
> + return 0;
> + }

Why does the core need to carry this quirk? If an accelerator does not
want to support interleaving then just don't ask for interleaved
capacity?

> +
> + guard(rwsem_read)(&cxl_region_rwsem);

See below...

> + if (ctx->host_bridge != cxlsd->target[0]->dport_dev) {
> + dev_dbg(dev, "host bridge does not match\n");
> + return 0;
> + }
> +
> + /*
> + * Walk the root decoder resource range relying on cxl_region_rwsem to
> + * preclude sibling arrival/departure and find the largest free space
> + * gap.
> + */
> + lockdep_assert_held_read(&cxl_region_rwsem);

The lock was just acquired a few lines up, no need for extra lockdep
assertion paranoia. However, I think the lock belongs outside of this
function otherwise the iterator of region is racing region creation.
However2, cxl_get_hpa_freespace() is already holding the lock!

So, I am not sure this code path has ever been tested as lockdep should
complain about the double acquisition.

> + max = 0;
> + res = cxlrd->res->child;
> +
> + /* With no resource child the whole parent resource is available */
> + if (!res)
> + max = resource_size(cxlrd->res);
> + else
> + max = 0;
> +
> + for (prev = NULL; res; prev = res, res = res->sibling) {
> + struct resource *next = res->sibling;
> + resource_size_t free = 0;
> +
> + /*
> + * Sanity check for preventing arithmetic problems below as a
> + * resource with size 0 could imply using the end field below
> + * when set to unsigned zero - 1 or all f in hex.
> + */
> + if (prev && !resource_size(prev))
> + continue;
> +
> + if (!prev && res->start > cxlrd->res->start) {
> + free = res->start - cxlrd->res->start;
> + max = max(free, max);
> + }
> + if (prev && res->start > prev->end + 1) {
> + free = res->start - prev->end + 1;
> + max = max(free, max);
> + }
> + if (next && res->end + 1 < next->start) {
> + free = next->start - res->end + 1;
> + max = max(free, max);
> + }
> + if (!next && res->end + 1 < cxlrd->res->end + 1) {
> + free = cxlrd->res->end + 1 - res->end + 1;
> + max = max(free, max);
> + }
> + }
> +
> + dev_dbg(CXLRD_DEV(cxlrd), "found %pa bytes of free space\n", &max);
> + if (max > ctx->max_hpa) {
> + if (ctx->cxlrd)
> + put_device(CXLRD_DEV(ctx->cxlrd));

What drove capitalizing "cxlrd_dev"?

> + get_device(CXLRD_DEV(cxlrd));
> + ctx->cxlrd = cxlrd;
> + ctx->max_hpa = max;
> + dev_dbg(CXLRD_DEV(cxlrd), "found %pa bytes of free space\n",
> + &max);
> + }
> + return 0;
> +}
> +
> +/**
> + * cxl_get_hpa_freespace - find a root decoder with free capacity per constraints
> + * @cxlmd: the CXL memory device with an endpoint that is mapped by the returned
> + * decoder
> + * @flags: CXL_DECODER_F flags for selecting RAM vs PMEM, and HDM-H vs HDM-D[B]
> + * @max_avail_contig: output parameter of max contiguous bytes available in the
> + * returned decoder
> + *
> + * The return tuple of a 'struct cxl_root_decoder' and 'bytes available given
> + * in (@max_avail_contig))' is a point in time snapshot. If by the time the
> + * caller goes to use this root decoder's capacity the capacity is reduced then
> + * caller needs to loop and retry.
> + *
> + * The returned root decoder has an elevated reference count that needs to be
> + * put with put_device(cxlrd_dev(cxlrd)). Locking context is with
> + * cxl_{acquire,release}_endpoint(), that ensures removal of the root decoder
> + * does not race.
> + */
> +struct cxl_root_decoder *cxl_get_hpa_freespace(struct cxl_memdev *cxlmd,
> + unsigned long flags,
> + resource_size_t *max_avail_contig)

I don't understand the rationale throwing away the ability to search
root decoders by additional constraints.

> +{
> + struct cxl_port *endpoint = cxlmd->endpoint;
> + struct cxlrd_max_context ctx = {
> + .host_bridge = endpoint->host_bridge,
> + .flags = flags,
> + };
> + struct cxl_port *root_port;
> + struct cxl_root *root __free(put_cxl_root) = find_cxl_root(endpoint);
> +
> + if (!is_cxl_endpoint(endpoint)) {
> + dev_dbg(&endpoint->dev, "hpa requestor is not an endpoint\n");
> + return ERR_PTR(-EINVAL);
> + }
> +
> + if (!root) {
> + dev_dbg(&endpoint->dev, "endpoint can not be related to a root port\n");
> + return ERR_PTR(-ENXIO);
> + }
> +
> + root_port = &root->port;
> + down_read(&cxl_region_rwsem);
> + device_for_each_child(&root_port->dev, &ctx, find_max_hpa);
> + up_read(&cxl_region_rwsem);
> +
> + if (!ctx.cxlrd)
> + return ERR_PTR(-ENOMEM);
> +
> + *max_avail_contig = ctx.max_hpa;
> + return ctx.cxlrd;
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_get_hpa_freespace, "CXL");

Lets just do EXPORT_SYMBOL_GPL() for any API that an accelerator would
use. The symbol namespace was more for warning about potential semantic
shortcuts and liberties taken by drivers/cxl/ modules talking to each
other. Anything that is exported for outside of drivers/cxl/ usage
should not take those liberties.

> +
> static ssize_t size_store(struct device *dev, struct device_attribute *attr,
> const char *buf, size_t len)
> {
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index a662b1b88408..efdd4627b774 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -785,6 +785,9 @@ static inline void cxl_dport_init_ras_reporting(struct cxl_dport *dport,
> struct cxl_decoder *to_cxl_decoder(struct device *dev);
> struct cxl_root_decoder *to_cxl_root_decoder(struct device *dev);
> struct cxl_switch_decoder *to_cxl_switch_decoder(struct device *dev);
> +
> +#define CXLRD_DEV(cxlrd) (&(cxlrd)->cxlsd.cxld.dev)

...oh, it's a macro now for some reason.

> +
> struct cxl_endpoint_decoder *to_cxl_endpoint_decoder(struct device *dev);
> bool is_root_decoder(struct device *dev);
> bool is_switch_decoder(struct device *dev);
> diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h
> index f7ce683465f0..4a8434a2b5da 100644
> --- a/include/cxl/cxl.h
> +++ b/include/cxl/cxl.h
> @@ -6,6 +6,10 @@
>
> #include <linux/ioport.h>
>
> +#define CXL_DECODER_F_RAM BIT(0)
> +#define CXL_DECODER_F_PMEM BIT(1)
> +#define CXL_DECODER_F_TYPE2 BIT(2)
> +
> enum cxl_resource {
> CXL_RES_DPA,
> CXL_RES_RAM,
> @@ -50,4 +54,8 @@ int cxl_release_resource(struct cxl_dev_state *cxlds, enum cxl_resource type);
> void cxl_set_media_ready(struct cxl_dev_state *cxlds);
> struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
> struct cxl_dev_state *cxlds);
> +struct cxl_port;
> +struct cxl_root_decoder *cxl_get_hpa_freespace(struct cxl_memdev *cxlmd,
> + unsigned long flags,
> + resource_size_t *max);

The name does not track for me, because nothing is acquired in this
function. It just surveys for a root decoder that meets the constraints.
It is possible that by the time the caller turns around to use that
freespace something else already grabbed it.