Re: [PATCH V14 0/7] CXL: Read CDAT

[Dan Williams <dan.j.williams@intel.com>]

Jonathan Cameron wrote:
> On Thu, 14 Jul 2022 20:04:17 -0700
> ira.weiny@intel.com wrote:
> 
> > From: Ira Weiny <ira.weiny@intel.com>
> > 
> > Details of changes are in the individual patches.
> > 
> > Major changes from V13:[10]
> > 	Dan minor updates
> > 	Willy's suggestion of documentation is good but I'm deferring it until
> > 	we get the location of the PCI mailboxes settled.
> > 	Drop retry CDAT patch
> > 	Drop DSMAS patch
> > 	Rebased on latest cxl-pending
> > 
> > CXL drivers need various data which are provided through generic DOE mailboxes
> > as defined in the PCIe 6.0 spec.[1]
> > 
> > One such data is the Coherent Device Attribute Table (CDAT).  CDAT data provides
> > coherent information about the various devices in the system.  It was developed
> > because systems no longer have a priori knowledge of all coherent devices
> > within a system.  CDAT describes the coherent characteristics of the
> > components on the CXL bus separate from system configurations.  The OS can
> > then, for example, use this information to form correct interleave sets.
> > 
> > To begin reading the CDAT the OS must have support to access the DOE mailboxes
> > provided by the CXL devices.
> > 
> > Because DOE is not specific to DOE but is provided within the PCI spec, the
> > series adds PCI DOE capability library functions.  These functions allow for
> > the iteration of the DOE capabilities on a device as well as creating
> > pci_doe_mb structures which can control the operation of the DOE state machine.
> > 
> > For now the iteration of and storage of the DOE mailboxes is done on memdev
> > objects within the CXL stack.  When this is needed in more generic code this
> > can be lifted later.
> > 
> > This work was tested using qemu.
> > 
> > [0] https://lore.kernel.org/linux-cxl/20211105235056.3711389-1-ira.weiny@intel.com/
> > [1] https://pcisig.com/specifications
> > [2] https://lore.kernel.org/qemu-devel/20210202005948.241655-1-ben.widawsky@intel.com/
> > [3] https://lore.kernel.org/linux-cxl/20220201071952.900068-1-ira.weiny@intel.com/
> > [4] https://lore.kernel.org/linux-cxl/20220330235920.2800929-1-ira.weiny@intel.com/
> > [5] https://lore.kernel.org/linux-cxl/20220414203237.2198665-1-ira.weiny@intel.com/
> > [6] https://lore.kernel.org/linux-cxl/20220531152632.1397976-1-ira.weiny@intel.com/
> > [7] https://lore.kernel.org/linux-cxl/20220605005049.2155874-1-ira.weiny@intel.com/
> > [8] https://lore.kernel.org/linux-cxl/20220610202259.3544623-1-ira.weiny@intel.com/
> > [9] https://lore.kernel.org/linux-cxl/20220628041527.742333-1-ira.weiny@intel.com/
> > [10] https://lore.kernel.org/linux-cxl/20220705154932.2141021-1-ira.weiny@intel.com/
> > 
> > 
> > Previous changes
> > ================
> > 
> > Changes from V12:[9]
> > 	A couple of bug fixes in the new XArray stuff
> > 	Remove the IRQ support because I did not realize how that worked and it
> > 	was complicating things.
> > 	Remove busy retries and replace with an error as there is no good way
> > 	to ensure it will work.
> 
> This is fine for userspace access, but I think we probably will want retries
> once we are using it in kernel.  Whilst we'd not expect it to be common as
> per (very late) reply I sent to v13 discussion, the CDAT table can change
> all on it's own (as far as software can see).  I'd expect it to be a once in
> a blue moon thing though.

It had better not change outside an explicit remap of the DPA space via
a command like set-partition with the immediate flag set... or maybe
after a firmware update. Anything is just unsupportable and broken and
the vendor of a device that changes the CDAT without the OS asking for
the change gets to keep the pieces.