Re: [PATCH 4/4] cxl: Make cxl_dpa_alloc() DPA partition number agnostic

[Dan Williams <dan.j.williams@intel.com>]

Alejandro Lucero Palau wrote:
> 
> On 1/17/25 20:57, Dan Williams wrote:
> > Alejandro Lucero Palau wrote:
> >> On 1/17/25 06:10, Dan Williams wrote:
> >>> cxl_dpa_alloc() is a hard coded nest of assumptions around PMEM
> >>> allocations being distinct from RAM allocations in specific ways when in
> >>> practice the allocation rules are only relative to DPA partition index.
> >>>
> >>> The rules for cxl_dpa_alloc() are:
> >>>
> >>> - allocations can only come from 1 partition
> >>>
> >>> - if allocating at partition-index-N, all free space in partitions less
> >>>     than partition-index-N must be skipped over
> >>
> >> In my view, you are mixing the current code with the new code in this
> >> explanation. It would be better to say the current code assumption is
> >> just two partitions, ram and pmem, but DCD changes the game.
> > There is no mixture in that description. The rules have not changed from
> > old to new, the implementation is updated to reflect that the algorithm
> > never needed to consider ram and pmem explicitly.
> 
> 
> Well, if I'm not mistaken, until now we did not need to support an 
> arbitrary number N of partitions but just 2.
> 
> In fact, your next sentence just confirms this.

Right, the algorithm can be more generic, and that is immediately needed
for the DCD case where it adds new partition types beyond pmem.

[..]
> > Holes are not allowed. If you want to delete any decoder capacity you
> > need to tear down all higher DPA allocations. That is a constraint of
> > the hardware definition around how software can assume the value of
> > DPABase. Will add some comments to that effect.
> 
> 
> I have to admit I do not master the case of regions created from user 
> space, but I thought the idea was to allow creation of regions from 
> independent devices for facilitating interleaving and just adding 
> flexibility when needing a memory region bigger than what only one 
> device can offer at some specific time.
> 
> 
> If so, I would expect cases like two different regions using different 
> ranges inside a device DPA, and the two regions completely independent. 
> If I understand your answer, this is possible but if you want to release 
> one of those regions, you can not use the released DPA until the first 
> one is also released ... my instinct tells me this can not be the case ...

As it turns out, the CXL specification mandates this counter-intuitive
situation. See the implementation note in the spec titled: "Device
Decode Logic". There you will see that if you have regionA and regionB
on a device using decoderA and decoderB where B is a higher DPA (and by
definition higher decoder id than A) you can not change the size of
decoderA without impacting the base address of decoderB. CXL HDM
Decoders are not PCI BARs.

> If what I'm saying makes no sense, I'll try with a more complex 
> description with a timeline and actions from user space, endpoint 
> decoders, regions and so on, what will definitely helps me to 
> (hopefully) understand the user space case.

See some of the commits that deal with this specification constraint:

105b6235ad0f cxl/port: Prevent out-of-order decoder allocation
101c268bd2f3 cxl/port: Fix use-after-free, permit out-of-order decoder shutdown
cb66b1d60c28 cxl/region: Allow out of order assembly of autodiscovered regions
2ab47045ac96 cxl/region: Flag partially torn down regions as unusable