[PATCH v4 00/25] dmaengine: Smart Data Accelerator Interface (SDXI) basic support

[PATCH v4 00/25] dmaengine: Smart Data Accelerator Interface (SDXI) basic support

[Nathan Lynch via B4 Relay]

The Smart Data Accelerator Interface (SDXI) is a vendor-neutral
architecture for memory-to-memory data movement offload designed for
kernel bypass and virtualization.

General information on SDXI may be found at:
https://www.snia.org/sdxi

This submission adds a driver with support for PCIe-hosted SDXI 1.0
implementations and includes a DMA engine provider with memcpy
capability.

Planned future SDXI work (out of scope for this series):

* Character device for exposing SDXI contexts to user space.

* Support for operation types to be added in future SDXI revisions.

* Greater configurability for control structures, e.g. descriptor ring
  size.

The latest released version of the SDXI specification is 1.0:
https://www.snia.org/sites/default/files/technical-work/sdxi/release/SNIA-SDXI-Specification-v1.0a.pdf

Starting with v4 the driver has been authored and modified with LLM
assistance. The code has changed a lot in response to Sashiko reviews and
it's been a long time since the last revision, so I dropped all
Reviewed-by: tags.

---
Changes in v4:
- Restructure the series to introduce helpers and resource management
  before their first users, with separate patches for function
  preparation, submission queues, and synchronized context release.
  Introduce the PCI remove callback earlier so SR-IOV cleanup is
  present from the initial PCI support. (Tycho Andersen)
- Rework function stopping to handle already-stopped functions without
  issuing a reset, wait for transient states, and escalate stalled
  soft stops to hard stops. Fall back to PCI function reset; if that
  fails, disable bus mastering and make a bounded attempt to drain
  pending transactions. (Sashiko)
- Sanitize inherited MMIO configuration before installing new control
  structures: disable error logging, discard stale context and RKey
  tables, clear error-log state, and disable supported mailbox
  interrupts.
- Stop/reset the function on activation failure and before freeing
  client contexts during removal. (Sashiko)
- Restore dma_set_mask_and_coherent() error checking and reject devices
  lacking the 64-bit atomic MMIO capability. (Sashiko for DMA mask
  error checking)
- Use the device's advertised maximum buffer size for context
  configuration and the DMA maximum segment size.
- Consolidate each submission queue's descriptor ring, write index,
  and context status into one DMA-coherent allocation.
- Replace per-operation completion allocations with a fixed,
  context-owned pool whose claims are independent of ring reuse.
  Retain timed-out completion blocks until context teardown rather
  than freeing memory the device may still access. (Sashiko)
- Rework completion checking to distinguish pending, successful, and
  failed operations. Read the signal with READ_ONCE() and endian
  conversion, place dma_rmb() after observing the terminal signal,
  and use read_poll_timeout() for polling. (Sashiko)
- Replace reservation iterators with indexed slot access, add
  variable-size reservations, and bound blocking reservations with
  a killable timeout. (Sashiko for iterator and unbounded-wait issues)
- Add sdxi_cxt_submit() to consolidate completion attachment, fencing,
  valid-bit publication, and optional doorbell ringing. Derive
  doorbell values from the current context write index.
- Add context-update descriptor encoding. Require a stopped context
  and complete update/sync before releasing its control structures;
  retain its resources and ID until function teardown if release
  fails. (Sashiko)
- Rework DMA preparation to store copy parameters outside the ring.
  Acquire completion claims and reserve ring space only when issuing,
  then populate every reserved slot. This avoids ring holes when
  prepared transactions are discarded. (Sashiko)
- Issue DMA transactions in cookie order, batch copies behind shared
  fenced completion interrupts, and queue work when ring or completion
  capacity is unavailable.
- Keep tx_status() read-only and retire transactions in cookie order
  through a shared IRQ/synchronization reaper. Report observed
  completion errors through tx_result as DMA_TRANS_ABORTED. (Sashiko)
- Rework termination to cancel submitted and pending work and reap
  published transactions without callbacks. Synchronize by polling
  and reaping transactions, then always flush virt-dma callbacks,
  eliminating the synchronization NOP reservation. (Sashiko)
- Initialize the AKey IDA before fallible context allocations, and
  sample the writable dma_channels parameter once during provider
  setup. (Sashiko)
- Expand KUnit coverage for completion-pool ownership, variable-size
  ring reservations, and context-update encoding. Fix the ring test's
  double advance and the context-stop test's incorrect union field.
  Suppress expected input-validation WARNs under KUnit and remove
  the dead size clamp. (Sashiko)
- Link to v3: https://patch.msgid.link/20260605-sdxi-base-v3-0-4d38ca2bdffe@amd.com

Changes in v3:
- Fix akey allocation error path in dma.c to return a proper
  error value. (Tycho Andersen)
- Disable SR-IOV in PCI removal. (TA)
- Update the Rust list of PCI class codes simultaneously with the C
  header. (Sashiko)
- Properly build the bus-agnostic core support as a separate
  module (sdxi-core) from the PCI driver (sdxi-pci). (Sashiko)
- Add dependency on CONFIG_64BIT to simplify assumptions around MMIO
  and control structure accesses. (Sashiko)
- Use readq/writeq instead of ioread64/iowrite64 since we don't need
  to handle port space. (Sashiko)
- Correct vector allocation range to ensure the error IRQ index (0) is
  reserved. (Sashiko)
- Fix context control block dma pool allocation failure
  check. (Sashiko)
- Ensure device is in stopped state before clearing MMIO_CTL0
  configuration during init. (Sashiko)
- Add explicit alignment attributes to packed control structure
  types. (Sashiko)
- Rename prep_memcpy_polled() to prep_memcpy_nointr(). (Frank Li)
- Link to v2: https://patch.msgid.link/20260511-sdxi-base-v2-0-889cfed17e3f@amd.com

Changes in v2:
- Drop unneeded dma_set_mask_and_coherent() result check. (Frank Li)
- Inline SDXI_DRV_DESC directly into MODULE_DESCRIPTION(). (FL)
- Drop unneeded braces from simple conditionals. (FL)
- Drop sdxi logging wrapper macros; use dev_dbg, dev_info etc
  directly. (FL)
- Reordering of commit message (patch 04, "Feature
  discovery..."). (FL)
- Use read_poll_timeout() for function start and stop routines. (FL)
- Align multi-line FIELD_PREP() uses. (FL)
- Drop sdxi_create_dma_pool() helper. (FL)
- Remove unneeded dma_wmb() before iowrite64() to doorbell. (FL)
- Use WRITE_ONCE() to update descriptor ring write index. (FL)
- Make sdxi_completion_poll() eventually time out and adjust call
  sites. (FL)
- Remove vestigial sdxi_dma_unregister() declaration. (FL)
- Reserve context ID before allocating context data structures instead
  of after.
- Update context ID class to transfer ownership of ID to context
  object; sdxi_free_cxt() now responsible for releasing ID once
  assigned.
- Align small frequently-updated DMA pool objects to cacheline
  boundaries.
- Drop redundant dma_set_mask_and_coherent() from DMA provider.
- Log unarchitected function status values in sdxi_dev_gsv().
- Remove sdxi_to_dev(); the abstraction is unnecessary and sdxi->dev
  is shorter.
- Link to v1: https://patch.msgid.link/20260410-sdxi-base-v1-0-1d184cb5c60a@amd.com

Changes in v1:
- Reorder series and introduce functionality incrementally while
  remaining buildable and functional at each step. (Jonathan Cameron)
- Use devres APIs where possible for device resources (JC)
- Use cleanup APIs to significantly reduce use of goto-oriented error
  unwinding. (JC)
- Drop SDXI_DEBUG config option. (JC)
- Cite SDXI spec version and section number consistently throughout. (JC)
- Combine local variable declarations of same type. (JC)
- Mark descriptor structs __packed. (JC)
- Use designated initializers in descriptor encoding functions. (JC)
- Prefer dev_err_probe() over sdxi_err() in sdxi_pci_init(). (JC)
- Prune unnecessary includes throughout source files. (JC)
- Remove unnecessary/unhelpful comments in several places. (JC)
- Remove SDXI spec material from "Add SNIA SDXI accelerator sub-class"
  commit message and reword the remainder. (Bjorn Helgaas)
- Remove unnecessary local for DMA_BIT_MASK() argument in
  sdxi_pci_init(). (BH)
- Use "{ }" for final null entry in id table, not "{ 0, }". (BH)
- Replace sample descriptor submission code from the SDXI spec with an
  improved API that has unit tests, eliminates a copy step for
  callers, and can block until ring space becomes available if
  desired.
- Omit the error log facility for now; it can be reintroduced later.
- Use a per-device xarray to allocate context IDs and map them to
  context objects.
- Implement interrupt-based completion signaling for memcpy operations
  in the DMA engine provider, DMA provider code mostly rewritten.

- Link to RFC: https://lore.kernel.org/r/20250905-sdxi-base-v1-0-d0341a1292ba@amd.com

---
Nathan Lynch (25):
      PCI: Add SNIA SDXI accelerator sub-class
      MAINTAINERS: Add entry for SDXI driver
      dmaengine: sdxi: Add PCI initialization
      dmaengine: sdxi: Prepare function and sanitize MMIO state
      dmaengine: sdxi: Feature discovery and initial configuration
      dmaengine: sdxi: Configure context tables
      dmaengine: sdxi: Allocate context control DMA pool
      dmaengine: sdxi: Introduce submission queue
      dmaengine: sdxi: Allocate administrative context
      dmaengine: sdxi: Install administrative context
      dmaengine: sdxi: Start functions on probe, stop on remove
      dmaengine: sdxi: Add client context allocation API
      dmaengine: sdxi: Add descriptor ring management
      dmaengine: sdxi: Add unit tests for descriptor ring reservations
      dmaengine: sdxi: Per-context access key (AKey) table entry allocator
      dmaengine: sdxi: Add completion status block API
      dmaengine: sdxi: Add unit tests for completion pool ownership
      dmaengine: sdxi: Add context submission helper
      dmaengine: sdxi: Encode context start, stop, update, and sync descriptors
      dmaengine: sdxi: Add client context release with update and sync
      dmaengine: sdxi: Provide context start and stop APIs
      dmaengine: sdxi: Encode copy and interrupt descriptors
      dmaengine: sdxi: Add unit tests for descriptor encoding
      dmaengine: sdxi: MSI/MSI-X vector allocation and mapping
      dmaengine: sdxi: Add DMA engine provider

 MAINTAINERS                         |   7 +
 drivers/dma/Kconfig                 |   2 +
 drivers/dma/Makefile                |   1 +
 drivers/dma/sdxi/.kunitconfig       |   4 +
 drivers/dma/sdxi/Kconfig            |  40 +++
 drivers/dma/sdxi/Makefile           |  17 ++
 drivers/dma/sdxi/completion.c       | 189 ++++++++++++
 drivers/dma/sdxi/completion.h       |  52 ++++
 drivers/dma/sdxi/completion_kunit.c | 131 ++++++++
 drivers/dma/sdxi/context.c          | 589 ++++++++++++++++++++++++++++++++++++
 drivers/dma/sdxi/context.h          | 122 ++++++++
 drivers/dma/sdxi/descriptor.c       | 220 ++++++++++++++
 drivers/dma/sdxi/descriptor.h       | 145 +++++++++
 drivers/dma/sdxi/descriptor_kunit.c | 537 ++++++++++++++++++++++++++++++++
 drivers/dma/sdxi/device.c           | 473 +++++++++++++++++++++++++++++
 drivers/dma/sdxi/dma.c              | 509 +++++++++++++++++++++++++++++++
 drivers/dma/sdxi/dma.h              |  11 +
 drivers/dma/sdxi/hw.h               | 257 ++++++++++++++++
 drivers/dma/sdxi/mmio.h             |  85 ++++++
 drivers/dma/sdxi/pci.c              | 153 ++++++++++
 drivers/dma/sdxi/ring.c             | 143 +++++++++
 drivers/dma/sdxi/ring.h             |  94 ++++++
 drivers/dma/sdxi/ring_kunit.c       | 147 +++++++++
 drivers/dma/sdxi/sdxi.h             | 138 +++++++++
 drivers/dma/sdxi/submission.h       |  66 ++++
 include/linux/pci_ids.h             |   1 +
 rust/kernel/pci/id.rs               |   1 +
 27 files changed, 4134 insertions(+)
---
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
change-id: 20250813-sdxi-base-73d7c9fdce57

Best regards,
--  
Nathan Lynch <nathan.lynch@amd.com>

[PATCH v4 01/25] PCI: Add SNIA SDXI accelerator sub-class

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Make the class code for SNIA Smart Data Accelerator Interface (SDXI)
functions available to both C and Rust code.

See PCI Code and ID Assignment spec r1.14, sec 1.19.

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 include/linux/pci_ids.h | 1 +
 rust/kernel/pci/id.rs   | 1 +
 2 files changed, 2 insertions(+)

diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index 030c86dd6d67..dcb26f52b5a1 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -154,6 +154,7 @@
 
 #define PCI_BASE_CLASS_ACCELERATOR	0x12
 #define PCI_CLASS_ACCELERATOR_PROCESSING	0x1200
+#define PCI_CLASS_ACCELERATOR_SDXI		0x120100
 
 #define PCI_CLASS_OTHERS		0xff
 
diff --git a/rust/kernel/pci/id.rs b/rust/kernel/pci/id.rs
index dbaf301666e7..d714e4b83852 100644
--- a/rust/kernel/pci/id.rs
+++ b/rust/kernel/pci/id.rs
@@ -292,6 +292,7 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
     SP_OTHER                   = bindings::PCI_CLASS_SP_OTHER,                   // 0x118000
 
     ACCELERATOR_PROCESSING     = bindings::PCI_CLASS_ACCELERATOR_PROCESSING,     // 0x120000
+    ACCELERATOR_SDXI           = bindings::PCI_CLASS_ACCELERATOR_SDXI,           // 0x120100
 
     OTHERS                     = bindings::PCI_CLASS_OTHERS,                     // 0xff0000
 }

-- 
2.56.0

[PATCH v4 02/25] MAINTAINERS: Add entry for SDXI driver

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Add an entry for the SDXI driver to MAINTAINERS. Wei and I will
maintain the driver.

The SDXI specification and other materials may be found at:

  https://www.snia.org/sdxi

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 MAINTAINERS | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 93abbb315065..721f06446435 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -25281,6 +25281,13 @@ L:	sdricohcs-devel@lists.sourceforge.net (subscribers-only)
 S:	Maintained
 F:	drivers/mmc/host/sdricoh_cs.c
 
+SDXI (Smart Data Accelerator Interface) DRIVER
+M:	Nathan Lynch <nathan.lynch@amd.com>
+M:	Wei Huang <wei.huang2@amd.com>
+L:	dmaengine@vger.kernel.org
+S:	Supported
+F:	drivers/dma/sdxi/
+
 SECO BOARDS CEC DRIVER
 M:	Ettore Chimenti <ek5.chimenti@gmail.com>
 S:	Maintained

-- 
2.56.0

[PATCH v4 03/25] dmaengine: sdxi: Add PCI initialization

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Add enough code to bind a SDXI device via the class code and map its
control registers and doorbell region.

Keep PCI resource setup in pci.c and register devices with the SDXI core
through bus operations.

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/Kconfig       |   2 +
 drivers/dma/Makefile      |   1 +
 drivers/dma/sdxi/Kconfig  |  28 +++++++++++++
 drivers/dma/sdxi/Makefile |   6 +++
 drivers/dma/sdxi/device.c |  33 +++++++++++++++
 drivers/dma/sdxi/mmio.h   |  22 ++++++++++
 drivers/dma/sdxi/pci.c    | 100 ++++++++++++++++++++++++++++++++++++++++++++++
 drivers/dma/sdxi/sdxi.h   |  38 ++++++++++++++++++
 8 files changed, 230 insertions(+)

diff --git a/drivers/dma/Kconfig b/drivers/dma/Kconfig
index 01f96a8257e5..3e3b685895fd 100644
--- a/drivers/dma/Kconfig
+++ b/drivers/dma/Kconfig
@@ -772,6 +772,8 @@ source "drivers/dma/lgm/Kconfig"
 
 source "drivers/dma/loongson/Kconfig"
 
+source "drivers/dma/sdxi/Kconfig"
+
 source "drivers/dma/stm32/Kconfig"
 
 # clients
diff --git a/drivers/dma/Makefile b/drivers/dma/Makefile
index f62d12b08e15..b375f9df9588 100644
--- a/drivers/dma/Makefile
+++ b/drivers/dma/Makefile
@@ -85,6 +85,7 @@ obj-$(CONFIG_XGENE_DMA) += xgene-dma.o
 obj-$(CONFIG_ST_FDMA) += st_fdma.o
 obj-$(CONFIG_FSL_DPAA2_QDMA) += fsl-dpaa2-qdma/
 obj-$(CONFIG_INTEL_LDMA) += lgm/
+obj-$(CONFIG_SDXI_CORE) += sdxi/
 
 obj-y += amd/
 obj-y += loongson/
diff --git a/drivers/dma/sdxi/Kconfig b/drivers/dma/sdxi/Kconfig
new file mode 100644
index 000000000000..b91b44231a04
--- /dev/null
+++ b/drivers/dma/sdxi/Kconfig
@@ -0,0 +1,28 @@
+config SDXI_CORE
+	tristate "SDXI (Smart Data Accelerator Interface) support"
+	# Doorbell updates require naturally-aligned 64-bit writes, per
+	# SDXI 1.0 9 MMIO Control Registers.
+	depends on 64BIT
+	select DMA_ENGINE
+	help
+	  Enable support for Smart Data Accelerator Interface (SDXI)
+	  Platform Data Mover devices. SDXI is a vendor-neutral
+	  standard for a memory-to-memory data mover and acceleration
+	  interface.
+
+	  This option is automatically selected by the SDXI PCI driver.
+	  Enable it manually if you want to build the SDXI core for unit
+	  testing without enabling the PCI driver.
+
+	  To compile this driver as a module, choose M here: the module
+	  will be named "sdxi-core".
+
+config SDXI_PCI
+	tristate "SDXI (Smart Data Accelerator Interface) PCI driver"
+	depends on PCI_MSI && 64BIT
+	select SDXI_CORE
+	help
+	  Enable support for PCI-hosted SDXI devices.
+
+	  To compile this driver as a module, choose M here: the module
+	  will be named "sdxi-pci".
diff --git a/drivers/dma/sdxi/Makefile b/drivers/dma/sdxi/Makefile
new file mode 100644
index 000000000000..0006edf74d86
--- /dev/null
+++ b/drivers/dma/sdxi/Makefile
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_SDXI_CORE) += sdxi-core.o
+sdxi-core-y := device.o
+
+obj-$(CONFIG_SDXI_PCI) += sdxi-pci.o
+sdxi-pci-y := pci.o
diff --git a/drivers/dma/sdxi/device.c b/drivers/dma/sdxi/device.c
new file mode 100644
index 000000000000..0974a83bb45c
--- /dev/null
+++ b/drivers/dma/sdxi/device.c
@@ -0,0 +1,33 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * SDXI hardware device driver
+ *
+ * Copyright Advanced Micro Devices, Inc.
+ */
+
+#include <linux/device.h>
+#include <linux/export.h>
+#include <linux/slab.h>
+
+#include "sdxi.h"
+
+int sdxi_register(struct device *dev, const struct sdxi_bus_ops *ops)
+{
+	struct sdxi_dev *sdxi;
+
+	sdxi = devm_kzalloc(dev, sizeof(*sdxi), GFP_KERNEL);
+	if (!sdxi)
+		return -ENOMEM;
+
+	sdxi->dev = dev;
+	sdxi->bus_ops = ops;
+	dev_set_drvdata(dev, sdxi);
+
+	return sdxi->bus_ops->init(sdxi);
+}
+EXPORT_SYMBOL_NS_GPL(sdxi_register, "SDXI");
+
+MODULE_AUTHOR("Wei Huang");
+MODULE_AUTHOR("Nathan Lynch");
+MODULE_DESCRIPTION("SDXI core");
+MODULE_LICENSE("GPL");
diff --git a/drivers/dma/sdxi/mmio.h b/drivers/dma/sdxi/mmio.h
new file mode 100644
index 000000000000..2676736f3998
--- /dev/null
+++ b/drivers/dma/sdxi/mmio.h
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+/*
+ * SDXI MMIO register offsets and layouts.
+ *
+ * Copyright Advanced Micro Devices, Inc.
+ */
+
+#ifndef DMA_SDXI_MMIO_H
+#define DMA_SDXI_MMIO_H
+
+#include <linux/bits.h>
+
+enum sdxi_reg {
+	/* SDXI 1.0 9.1 General Control and Status Registers */
+	SDXI_MMIO_CAP1       = 0x00208,
+};
+
+/* SDXI 1.0 Table 9-7: MMIO_CAP1 */
+#define SDXI_MMIO_CAP1_MMIO64 BIT_ULL(6)
+
+#endif  /* DMA_SDXI_MMIO_H */
diff --git a/drivers/dma/sdxi/pci.c b/drivers/dma/sdxi/pci.c
new file mode 100644
index 000000000000..0778cde31f14
--- /dev/null
+++ b/drivers/dma/sdxi/pci.c
@@ -0,0 +1,100 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * SDXI PCI device code
+ *
+ * Copyright Advanced Micro Devices, Inc.
+ */
+
+#include <linux/bitfield.h>
+#include <linux/dev_printk.h>
+#include <linux/dma-mapping.h>
+#include <linux/err.h>
+#include <linux/io.h>
+#include <linux/iomap.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+
+#include "mmio.h"
+#include "sdxi.h"
+
+enum sdxi_mmio_bars {
+	SDXI_PCI_BAR_CTL_REGS = 0,
+	SDXI_PCI_BAR_DOORBELL = 2,
+};
+
+static struct pci_dev *sdxi_to_pci_dev(const struct sdxi_dev *sdxi)
+{
+	return to_pci_dev(sdxi->dev);
+}
+
+static int sdxi_pci_init(struct sdxi_dev *sdxi)
+{
+	struct pci_dev *pdev = sdxi_to_pci_dev(sdxi);
+	struct device *dev = &pdev->dev;
+	u64 cap1;
+	int ret;
+
+	ret = pcim_enable_device(pdev);
+	if (ret)
+		return dev_err_probe(dev, ret, "failed to enable device\n");
+
+	ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64));
+	if (ret)
+		return dev_err_probe(dev, ret, "failed to set DMA mask\n");
+
+	sdxi->ctrl_regs = pcim_iomap_region(pdev, SDXI_PCI_BAR_CTL_REGS,
+					    KBUILD_MODNAME);
+	if (IS_ERR(sdxi->ctrl_regs))
+		return dev_err_probe(dev, PTR_ERR(sdxi->ctrl_regs),
+				     "failed to map control registers\n");
+
+	cap1 = readq(sdxi->ctrl_regs + SDXI_MMIO_CAP1);
+	if (!FIELD_GET(SDXI_MMIO_CAP1_MMIO64, cap1))
+		return dev_err_probe(dev, -EOPNOTSUPP,
+				     "64-bit atomic MMIO access required\n");
+
+	sdxi->dbs = pcim_iomap_region(pdev, SDXI_PCI_BAR_DOORBELL,
+				      KBUILD_MODNAME);
+	if (IS_ERR(sdxi->dbs))
+		return dev_err_probe(dev, PTR_ERR(sdxi->dbs),
+				     "failed to map doorbell region\n");
+
+	pci_set_master(pdev);
+	return 0;
+}
+
+static const struct sdxi_bus_ops sdxi_pci_ops = {
+	.init = sdxi_pci_init,
+};
+
+static int sdxi_pci_probe(struct pci_dev *pdev,
+			  const struct pci_device_id *id)
+{
+	return sdxi_register(&pdev->dev, &sdxi_pci_ops);
+}
+
+static void sdxi_pci_remove(struct pci_dev *pdev)
+{
+	pci_disable_sriov(pdev);
+}
+
+static const struct pci_device_id sdxi_id_table[] = {
+	{ PCI_DEVICE_CLASS(PCI_CLASS_ACCELERATOR_SDXI, 0xffffff) },
+	{ }
+};
+MODULE_DEVICE_TABLE(pci, sdxi_id_table);
+
+static struct pci_driver sdxi_driver = {
+	.name = "sdxi",
+	.id_table = sdxi_id_table,
+	.probe = sdxi_pci_probe,
+	.remove = sdxi_pci_remove,
+	.sriov_configure = pci_sriov_configure_simple,
+};
+
+MODULE_IMPORT_NS("SDXI");
+MODULE_AUTHOR("Wei Huang");
+MODULE_AUTHOR("Nathan Lynch");
+MODULE_DESCRIPTION("SDXI PCIe interface driver");
+MODULE_LICENSE("GPL");
+module_pci_driver(sdxi_driver);
diff --git a/drivers/dma/sdxi/sdxi.h b/drivers/dma/sdxi/sdxi.h
new file mode 100644
index 000000000000..d4c61ca2f875
--- /dev/null
+++ b/drivers/dma/sdxi/sdxi.h
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * SDXI device driver header
+ *
+ * Copyright Advanced Micro Devices, Inc.
+ */
+
+#ifndef DMA_SDXI_H
+#define DMA_SDXI_H
+
+#include <linux/compiler_types.h>
+#include <linux/types.h>
+
+struct sdxi_dev;
+
+/**
+ * struct sdxi_bus_ops - Bus-specific methods for SDXI devices.
+ */
+struct sdxi_bus_ops {
+	/**
+	 * @init: Map control registers and doorbell region, allocate
+	 *        IRQ ranges. Invoked before bus-agnostic SDXI
+	 *        function initialization.
+	 */
+	int (*init)(struct sdxi_dev *sdxi);
+};
+
+struct sdxi_dev {
+	struct device *dev;
+	void __iomem *ctrl_regs;	/* virt addr of ctrl registers */
+	void __iomem *dbs;		/* virt addr of doorbells */
+
+	const struct sdxi_bus_ops *bus_ops;
+};
+
+int sdxi_register(struct device *dev, const struct sdxi_bus_ops *ops);
+
+#endif /* DMA_SDXI_H */

-- 
2.56.0

[PATCH v4 04/25] dmaengine: sdxi: Prepare function and sanitize MMIO state

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Recover an inherited function state using bounded native stop/reset
transitions and a PCI function reset that saves and restores PCI
configuration. Use the locked reset API because probe and removal hold
the device lock. If reset fails, disable bus mastering and wait
for pending transactions as a best-effort fallback.

Disable inherited error logging before stop commands can provoke errors.
After stopping, sanitize writable configuration in Chapter 9 order:
preserve the function group ID, clear PASID and interrupt enables,
restore CTL2 reset limits with operation groups disabled, discard stale
context and supported RKey tables, and clear error-log state. Disable
mailbox interrupts only on functions with SR-IOV mailbox support.

Keep mailbox commands and peer-owned payloads untouched. Status cleanup
uses the specified write-one-to-clear semantics. Invoke preparation
before feature configuration or publishing new DMA control structures.

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/device.c | 271 +++++++++++++++++++++++++++++++++++++++++++++-
 drivers/dma/sdxi/mmio.h   |  49 ++++++++-
 drivers/dma/sdxi/pci.c    |  24 ++++
 drivers/dma/sdxi/sdxi.h   |  20 ++++
 4 files changed, 362 insertions(+), 2 deletions(-)

diff --git a/drivers/dma/sdxi/device.c b/drivers/dma/sdxi/device.c
index 0974a83bb45c..23a75beb6912 100644
--- a/drivers/dma/sdxi/device.c
+++ b/drivers/dma/sdxi/device.c
@@ -5,15 +5,280 @@
  * Copyright Advanced Micro Devices, Inc.
  */
 
+#include <linux/bitfield.h>
+#include <linux/delay.h>
 #include <linux/device.h>
 #include <linux/export.h>
+#include <linux/iopoll.h>
+#include <linux/jiffies.h>
 #include <linux/slab.h>
+#include <linux/time.h>
 
+#include "mmio.h"
 #include "sdxi.h"
 
+enum sdxi_fn_gsv {
+	SDXI_GSV_STOP     = 0,
+	SDXI_GSV_INIT     = 1,
+	SDXI_GSV_ACTIVE   = 2,
+	SDXI_GSV_STOPG_SF = 3,
+	SDXI_GSV_STOPG_HD = 4,
+	SDXI_GSV_ERROR    = 5,
+};
+
+static const char *const gsv_strings[] = {
+	[SDXI_GSV_STOP]     = "stopped",
+	[SDXI_GSV_INIT]     = "initializing",
+	[SDXI_GSV_ACTIVE]   = "active",
+	[SDXI_GSV_STOPG_SF] = "soft stopping",
+	[SDXI_GSV_STOPG_HD] = "hard stopping",
+	[SDXI_GSV_ERROR]    = "error",
+};
+
+static const char *gsv_str(enum sdxi_fn_gsv gsv)
+{
+	if ((size_t)gsv < ARRAY_SIZE(gsv_strings))
+		return gsv_strings[(size_t)gsv];
+
+	return "unknown";
+}
+
+enum sdxi_fn_gsr {
+	SDXI_GSRV_RESET   = 0,
+	SDXI_GSRV_STOP_SF = 1,
+	SDXI_GSRV_STOP_HD = 2,
+	SDXI_GSRV_ACTIVE  = 3,
+};
+
+static const char *const gsr_strings[] = {
+	[SDXI_GSRV_RESET]   = "reset",
+	[SDXI_GSRV_STOP_SF] = "soft stop",
+	[SDXI_GSRV_STOP_HD] = "hard stop",
+	[SDXI_GSRV_ACTIVE]  = "activate",
+};
+
+static const char *gsr_str(enum sdxi_fn_gsr gsr)
+{
+	if ((size_t)gsr < ARRAY_SIZE(gsr_strings))
+		return gsr_strings[(size_t)gsr];
+
+	return "unknown";
+}
+
+static enum sdxi_fn_gsv sdxi_dev_gsv(const struct sdxi_dev *sdxi)
+{
+	u64 sts0 = sdxi_read64(sdxi, SDXI_MMIO_STS0);
+	enum sdxi_fn_gsv gsv = FIELD_GET(SDXI_MMIO_STS0_FN_GSV, sts0);
+
+	dev_dbg(sdxi->dev, "MMIO_STS0.fn_gsv = %#x (%s)\n", gsv, gsv_str(gsv));
+
+	switch (gsv) {
+	case SDXI_GSV_STOP ... SDXI_GSV_ERROR:
+		break;
+	default:
+		dev_warn_ratelimited(sdxi->dev, "unknown gsv %u\n", gsv);
+		break;
+	}
+
+	return gsv;
+}
+
+static const unsigned long gsv_poll_interval_us = USEC_PER_MSEC * 100;
+static const unsigned long gsv_transition_timeout_us = USEC_PER_SEC * 1;
+
+#define sdxi_dev_gsv_poll(sdxi, val, cond)				\
+	read_poll_timeout(sdxi_dev_gsv, val, cond, gsv_poll_interval_us, \
+			  gsv_transition_timeout_us, false, sdxi)
+
+static void sdxi_write_fn_gsr(struct sdxi_dev *sdxi, enum sdxi_fn_gsr cmd)
+{
+	u64 ctl0 = sdxi_read64(sdxi, SDXI_MMIO_CTL0);
+
+	dev_dbg(sdxi->dev, "MMIO_CTL0.fn_gsr = %#x (%s)\n", cmd, gsr_str(cmd));
+
+	FIELD_MODIFY(SDXI_MMIO_CTL0_FN_GSR, &ctl0, cmd);
+	sdxi_write64(sdxi, SDXI_MMIO_CTL0, ctl0);
+}
+
+/*
+ * How to make progress toward GSV_STOP from each function state. See
+ * SDXI 1.0 Figure 4-1: SDXI Function States and State Transitions.
+ *
+ * The function ordinarily leaves a @transient state without software
+ * intervention, so @cmd applies only to the others. @stall_cmd, if any,
+ * is issued when the state outlasts the poll timeout. @next is the set
+ * of states software may observe after the step.
+ */
+struct gsv_stop_step {
+	bool transient:1;
+	bool issue_stall_cmd:1;
+	enum sdxi_fn_gsr cmd;
+	enum sdxi_fn_gsr stall_cmd;
+	unsigned long next;
+};
+
+static const struct gsv_stop_step gsv_stop_steps[] = {
+	[SDXI_GSV_STOP] = { },
+	[SDXI_GSV_INIT] = {
+		/*
+		 * Any fn_gsr write other than SDXI_GSRV_ACTIVE during
+		 * initialization forces HaltErr:Fn -> GSV_ERROR.
+		 * GSV_STOPG_HD and GSV_STOP are valid next values since
+		 * the stop command could race with a GSV_INIT ->
+		 * GSV_ACTIVE transition.
+		 */
+		.transient = true,
+		.issue_stall_cmd = true,
+		.stall_cmd = SDXI_GSRV_STOP_HD,
+		.next = BIT(SDXI_GSV_ACTIVE) | BIT(SDXI_GSV_ERROR) |
+			BIT(SDXI_GSV_STOPG_HD) | BIT(SDXI_GSV_STOP),
+	},
+	[SDXI_GSV_ACTIVE] = {
+		.cmd = SDXI_GSRV_STOP_SF,
+		.next = BIT(SDXI_GSV_STOPG_SF) | BIT(SDXI_GSV_STOP) |
+			BIT(SDXI_GSV_ERROR),
+	},
+	[SDXI_GSV_STOPG_SF] = {
+		.transient = true,
+		.issue_stall_cmd = true,
+		.stall_cmd = SDXI_GSRV_STOP_HD,
+		.next = BIT(SDXI_GSV_STOPG_HD) | BIT(SDXI_GSV_STOP) |
+			BIT(SDXI_GSV_ERROR),
+	},
+	[SDXI_GSV_STOPG_HD] = {
+		/* All fn_gsr writes are ignored here; only waiting helps. */
+		.transient = true,
+		.next = BIT(SDXI_GSV_STOP) | BIT(SDXI_GSV_ERROR),
+	},
+	[SDXI_GSV_ERROR] = {
+		.cmd = SDXI_GSRV_RESET,
+		.next = BIT(SDXI_GSV_STOP),
+	},
+};
+
+/* Drive the function to GSV_STOP from any state. */
+static int sdxi_dev_stop(struct sdxi_dev *sdxi)
+{
+	unsigned long stepped_from = 0;
+	enum sdxi_fn_gsv gsv;
+
+	/*
+	 * Each step must reach a state we have not already stepped from.
+	 * The table admits no cycles, so revisiting one means the device
+	 * is not behaving as specified.
+	 */
+	for (gsv = sdxi_dev_gsv(sdxi); gsv != SDXI_GSV_STOP; ) {
+		const struct gsv_stop_step *step;
+		enum sdxi_fn_gsv from = gsv;
+
+		if ((size_t)gsv >= ARRAY_SIZE(gsv_stop_steps))
+			break;
+		if (stepped_from & BIT(gsv))
+			break;
+		stepped_from |= BIT(gsv);
+
+		step = &gsv_stop_steps[gsv];
+		if (!step->transient)
+			sdxi_write_fn_gsr(sdxi, step->cmd);
+
+		if (sdxi_dev_gsv_poll(sdxi, gsv, gsv != from)) {
+			if (!step->issue_stall_cmd)
+				break;
+			dev_notice(sdxi->dev, "stop: %s, issuing %s\n",
+				   gsv_str(from), gsr_str(step->stall_cmd));
+			sdxi_write_fn_gsr(sdxi, step->stall_cmd);
+			if (sdxi_dev_gsv_poll(sdxi, gsv, gsv != from))
+				break;
+		}
+
+		WARN_ONCE(!(BIT(gsv) & step->next), "unexpected %s -> %s\n",
+			  gsv_str(from), gsv_str(gsv));
+	}
+
+	if (gsv == SDXI_GSV_STOP)
+		return 0;
+
+	dev_err(sdxi->dev, "stop: stuck at MMIO_STS0.fn_gsv = %#x (%s)\n",
+		gsv, gsv_str(gsv));
+
+	return sdxi->bus_ops->reset ? sdxi->bus_ops->reset(sdxi) : -EIO;
+}
+
+/* Stop the function and discard inherited MMIO configuration. */
+static int sdxi_fn_prepare(struct sdxi_dev *sdxi)
+{
+	u64 cap1, ctl0, ctl2, err_cfg, rkey;
+	int err;
+
+	/* Disable inherited error logging before stop can provoke an error. */
+	err_cfg = sdxi_read64(sdxi, SDXI_MMIO_ERR_CFG);
+	FIELD_MODIFY(SDXI_MMIO_ERR_CFG_EN, &err_cfg, 0);
+	sdxi_write64(sdxi, SDXI_MMIO_ERR_CFG, err_cfg);
+	err_cfg = sdxi_read64(sdxi, SDXI_MMIO_ERR_CFG);
+	if (FIELD_GET(SDXI_MMIO_ERR_CFG_EN, err_cfg)) {
+		dev_err(sdxi->dev, "failed to disable error logging\n");
+		return -EIO;
+	}
+
+	/*
+	 * Ensure the function is in GSV_STOP state, then clear ctl0's
+	 * pasid and error interrupt configuration while preserving
+	 * any assigned group ID (fn_grp_id).
+	 */
+	err = sdxi_dev_stop(sdxi);
+	if (err)
+		return err;
+
+	ctl0 = sdxi_read64(sdxi, SDXI_MMIO_CTL0);
+	FIELD_MODIFY(SDXI_MMIO_CTL0_FN_ERR_INTR_EN, &ctl0, 0);
+	FIELD_MODIFY(SDXI_MMIO_CTL0_FN_PASID_VL, &ctl0, 0);
+	FIELD_MODIFY(SDXI_MMIO_CTL0_FN_PASID, &ctl0, 0);
+	sdxi_write64(sdxi, SDXI_MMIO_CTL0, ctl0);
+
+	/* Restore CTL2's reset limits, with operation groups disabled. */
+	cap1 = sdxi_read64(sdxi, SDXI_MMIO_CAP1);
+	ctl2 = FIELD_PREP(SDXI_MMIO_CTL2_MAX_BUFFER,
+			  FIELD_GET(SDXI_MMIO_CAP1_MAX_BUFFER, cap1));
+	ctl2 |= FIELD_PREP(SDXI_MMIO_CTL2_MAX_AKEY_SZ,
+			   FIELD_GET(SDXI_MMIO_CAP1_MAX_AKEY_SZ, cap1));
+	ctl2 |= FIELD_PREP(SDXI_MMIO_CTL2_MAX_CXT,
+			   FIELD_GET(SDXI_MMIO_CAP1_MAX_CXT, cap1));
+	sdxi_write64(sdxi, SDXI_MMIO_CTL2, ctl2);
+
+	/* No inherited context or remote-access tables are owned by us. */
+	sdxi_write64(sdxi, SDXI_MMIO_CXT_L2, 0);
+	if (FIELD_GET(SDXI_MMIO_CAP1_RKEY_CAP, cap1)) {
+		rkey = sdxi_read64(sdxi, SDXI_MMIO_RKEY);
+		FIELD_MODIFY(SDXI_MMIO_RKEY_EN, &rkey, 0);
+		sdxi_write64(sdxi, SDXI_MMIO_RKEY, rkey);
+		rkey = sdxi_read64(sdxi, SDXI_MMIO_RKEY);
+		if (FIELD_GET(SDXI_MMIO_RKEY_EN, rkey)) {
+			dev_err(sdxi->dev, "failed to disable RKey access\n");
+			return -EIO;
+		}
+		sdxi_write64(sdxi, SDXI_MMIO_RKEY, 0);
+	}
+
+	/* Clear error-log state using the status register's RW1C semantics. */
+	sdxi_write64(sdxi, SDXI_MMIO_ERR_CTL, 0);
+	sdxi_write64(sdxi, SDXI_MMIO_ERR_STS,
+		     SDXI_MMIO_ERR_STS_STS_BIT | SDXI_MMIO_ERR_STS_OVF_BIT |
+		     SDXI_MMIO_ERR_STS_ERR_BIT);
+	sdxi_write64(sdxi, SDXI_MMIO_ERR_CFG, 0);
+	sdxi_write64(sdxi, SDXI_MMIO_ERR_WRT, 0);
+	sdxi_write64(sdxi, SDXI_MMIO_ERR_RD, 0);
+
+	/* Mailbox contents and commands belong to the peer protocol. */
+	if (sdxi->has_mailbox)
+		sdxi_write64(sdxi, SDXI_MMIO_MB_CTL, 0);
+
+	return 0;
+}
+
 int sdxi_register(struct device *dev, const struct sdxi_bus_ops *ops)
 {
 	struct sdxi_dev *sdxi;
+	int err;
 
 	sdxi = devm_kzalloc(dev, sizeof(*sdxi), GFP_KERNEL);
 	if (!sdxi)
@@ -23,7 +288,11 @@ int sdxi_register(struct device *dev, const struct sdxi_bus_ops *ops)
 	sdxi->bus_ops = ops;
 	dev_set_drvdata(dev, sdxi);
 
-	return sdxi->bus_ops->init(sdxi);
+	err = sdxi->bus_ops->init(sdxi);
+	if (err)
+		return err;
+
+	return sdxi_fn_prepare(sdxi);
 }
 EXPORT_SYMBOL_NS_GPL(sdxi_register, "SDXI");
 
diff --git a/drivers/dma/sdxi/mmio.h b/drivers/dma/sdxi/mmio.h
index 2676736f3998..a9c1fbb404d5 100644
--- a/drivers/dma/sdxi/mmio.h
+++ b/drivers/dma/sdxi/mmio.h
@@ -13,10 +13,57 @@
 
 enum sdxi_reg {
 	/* SDXI 1.0 9.1 General Control and Status Registers */
+	SDXI_MMIO_CTL0       = 0x00000,
+	SDXI_MMIO_CTL2       = 0x00010,
+	SDXI_MMIO_STS0       = 0x00100,
 	SDXI_MMIO_CAP1       = 0x00208,
+
+	/* SDXI 1.0 9.2 Context and RKey Table Registers */
+	SDXI_MMIO_CXT_L2     = 0x10000,
+	SDXI_MMIO_RKEY       = 0x10100,
+
+	/* SDXI 1.0 9.3 Error Logging Control and Status Registers */
+	SDXI_MMIO_ERR_CTL    = 0x20000,
+	SDXI_MMIO_ERR_STS    = 0x20008,
+	SDXI_MMIO_ERR_CFG    = 0x20010,
+	SDXI_MMIO_ERR_WRT    = 0x20020,
+	SDXI_MMIO_ERR_RD     = 0x20028,
+
+	/* SDXI 1.0 9.4 MBOX Mailbox Registers */
+	SDXI_MMIO_MB_CTL     = 0x30000,
 };
 
+/* SDXI 1.0 Table 9-2: MMIO_CTL0 */
+#define SDXI_MMIO_CTL0_FN_GSR         GENMASK_ULL(1, 0)
+#define SDXI_MMIO_CTL0_FN_PASID_VL    BIT_ULL(2)
+#define SDXI_MMIO_CTL0_FN_ERR_INTR_EN BIT_ULL(4)
+#define SDXI_MMIO_CTL0_FN_PASID       GENMASK_ULL(27, 8)
+
+/* SDXI 1.0 Table 9-4: MMIO_CTL2 */
+#define SDXI_MMIO_CTL2_MAX_BUFFER  GENMASK_ULL(3, 0)
+#define SDXI_MMIO_CTL2_MAX_AKEY_SZ GENMASK_ULL(15, 12)
+#define SDXI_MMIO_CTL2_MAX_CXT     GENMASK_ULL(31, 16)
+#define SDXI_MMIO_CTL2_OPB_000_AVL GENMASK_ULL(63, 32)
+
+/* SDXI 1.0 Table 9-5: MMIO_STS0 */
+#define SDXI_MMIO_STS0_FN_GSV GENMASK_ULL(2, 0)
+
 /* SDXI 1.0 Table 9-7: MMIO_CAP1 */
-#define SDXI_MMIO_CAP1_MMIO64 BIT_ULL(6)
+#define SDXI_MMIO_CAP1_MAX_BUFFER  GENMASK_ULL(3, 0)
+#define SDXI_MMIO_CAP1_RKEY_CAP    BIT_ULL(4)
+#define SDXI_MMIO_CAP1_MMIO64      BIT_ULL(6)
+#define SDXI_MMIO_CAP1_MAX_AKEY_SZ GENMASK_ULL(15, 12)
+#define SDXI_MMIO_CAP1_MAX_CXT     GENMASK_ULL(31, 16)
+
+/* SDXI 1.0 Table 9-10: MMIO_RKEY */
+#define SDXI_MMIO_RKEY_EN BIT_ULL(0)
+
+/* SDXI 1.0 Table 9-12: MMIO_ERR_STS (write one to clear) */
+#define SDXI_MMIO_ERR_STS_STS_BIT BIT_ULL(0)
+#define SDXI_MMIO_ERR_STS_OVF_BIT BIT_ULL(1)
+#define SDXI_MMIO_ERR_STS_ERR_BIT BIT_ULL(3)
+
+/* SDXI 1.0 Table 9-13: MMIO_ERR_CFG */
+#define SDXI_MMIO_ERR_CFG_EN BIT_ULL(0)
 
 #endif  /* DMA_SDXI_MMIO_H */
diff --git a/drivers/dma/sdxi/pci.c b/drivers/dma/sdxi/pci.c
index 0778cde31f14..6a293db69199 100644
--- a/drivers/dma/sdxi/pci.c
+++ b/drivers/dma/sdxi/pci.c
@@ -59,12 +59,36 @@ static int sdxi_pci_init(struct sdxi_dev *sdxi)
 		return dev_err_probe(dev, PTR_ERR(sdxi->dbs),
 				     "failed to map doorbell region\n");
 
+	/* Mailbox registers are reserved without SR-IOV support. */
+	sdxi->has_mailbox = pdev->is_virtfn ||
+		pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_SRIOV);
+
 	pci_set_master(pdev);
 	return 0;
 }
 
+static int sdxi_pci_reset(struct sdxi_dev *sdxi)
+{
+	struct pci_dev *pdev = sdxi_to_pci_dev(sdxi);
+	int err;
+
+	/* Probe and remove hold the device lock; preserve PCI configuration. */
+	err = pci_reset_function_locked(pdev);
+	if (err) {
+		dev_err(&pdev->dev,
+			"function reset failed (%d), disabling bus mastering\n", err);
+		pci_clear_master(pdev);
+		if (!pci_wait_for_pending_transaction(pdev))
+			dev_warn(&pdev->dev,
+				 "PCIe transactions still pending after disabling bus mastering\n");
+	}
+
+	return err;
+}
+
 static const struct sdxi_bus_ops sdxi_pci_ops = {
 	.init = sdxi_pci_init,
+	.reset = sdxi_pci_reset,
 };
 
 static int sdxi_pci_probe(struct pci_dev *pdev,
diff --git a/drivers/dma/sdxi/sdxi.h b/drivers/dma/sdxi/sdxi.h
index d4c61ca2f875..bb0d2165d495 100644
--- a/drivers/dma/sdxi/sdxi.h
+++ b/drivers/dma/sdxi/sdxi.h
@@ -9,8 +9,12 @@
 #define DMA_SDXI_H
 
 #include <linux/compiler_types.h>
+#include <linux/dev_printk.h>
+#include <linux/io.h>
 #include <linux/types.h>
 
+#include "mmio.h"
+
 struct sdxi_dev;
 
 /**
@@ -23,6 +27,10 @@ struct sdxi_bus_ops {
 	 *        function initialization.
 	 */
 	int (*init)(struct sdxi_dev *sdxi);
+	/**
+	 * @reset: Bus-specific recovery when the native stop sequence fails.
+	 */
+	int (*reset)(struct sdxi_dev *sdxi);
 };
 
 struct sdxi_dev {
@@ -30,9 +38,21 @@ struct sdxi_dev {
 	void __iomem *ctrl_regs;	/* virt addr of ctrl registers */
 	void __iomem *dbs;		/* virt addr of doorbells */
 
+	bool has_mailbox;
+
 	const struct sdxi_bus_ops *bus_ops;
 };
 
 int sdxi_register(struct device *dev, const struct sdxi_bus_ops *ops);
 
+static inline u64 sdxi_read64(const struct sdxi_dev *sdxi, enum sdxi_reg reg)
+{
+	return readq(sdxi->ctrl_regs + reg);
+}
+
+static inline void sdxi_write64(struct sdxi_dev *sdxi, enum sdxi_reg reg, u64 val)
+{
+	writeq(val, sdxi->ctrl_regs + reg);
+}
+
 #endif /* DMA_SDXI_H */

-- 
2.56.0

[PATCH v4 05/25] dmaengine: sdxi: Feature discovery and initial configuration

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Discover the stopped function's version, doorbell stride, supported
operation groups, and context and buffer limits. Apply the driver
configuration to MMIO_CTL2 after function preparation has removed
inherited state.

Add capability fields and device bookkeeping. Report the SDXI version implemented by the function.

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/device.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++-
 drivers/dma/sdxi/mmio.h   | 13 +++++++++++++
 drivers/dma/sdxi/sdxi.h   |  6 ++++++
 3 files changed, 65 insertions(+), 1 deletion(-)

diff --git a/drivers/dma/sdxi/device.c b/drivers/dma/sdxi/device.c
index 23a75beb6912..c03af5833414 100644
--- a/drivers/dma/sdxi/device.c
+++ b/drivers/dma/sdxi/device.c
@@ -8,9 +8,13 @@
 #include <linux/bitfield.h>
 #include <linux/delay.h>
 #include <linux/device.h>
+#include <linux/dma-mapping.h>
 #include <linux/export.h>
 #include <linux/iopoll.h>
 #include <linux/jiffies.h>
+#include <linux/limits.h>
+#include <linux/minmax.h>
+#include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/time.h>
 
@@ -275,6 +279,43 @@ static int sdxi_fn_prepare(struct sdxi_dev *sdxi)
 	return 0;
 }
 
+/*
+ * See SDXI 1.0 4.1.8 Activation of the SDXI Function by Software.
+ */
+static int sdxi_fn_activate(struct sdxi_dev *sdxi)
+{
+	u64 version, cap0, cap1, ctl2;
+
+	version = sdxi_read64(sdxi, SDXI_MMIO_VERSION);
+	dev_info(sdxi->dev, "SDXI %llu.%llu device found\n",
+		  FIELD_GET(SDXI_MMIO_VERSION_MAJOR, version),
+		  FIELD_GET(SDXI_MMIO_VERSION_MINOR, version));
+
+	/* Read capabilities and features. */
+	cap0 = sdxi_read64(sdxi, SDXI_MMIO_CAP0);
+	sdxi->db_stride = SZ_4K;
+	sdxi->db_stride *= 1U << FIELD_GET(SDXI_MMIO_CAP0_DB_STRIDE, cap0);
+
+	cap1 = sdxi_read64(sdxi, SDXI_MMIO_CAP1);
+	sdxi->op_grp_cap = FIELD_GET(SDXI_MMIO_CAP1_OPB_000_CAP, cap1);
+	sdxi->max_buffer = FIELD_GET(SDXI_MMIO_CAP1_MAX_BUFFER, cap1);
+	dma_set_max_seg_size(sdxi->dev,
+			     min((u64)SZ_2M << sdxi->max_buffer, U32_MAX));
+	sdxi->max_cxtid = FIELD_GET(SDXI_MMIO_CAP1_MAX_CXT, cap1);
+
+	/* Apply our configuration. */
+	ctl2 = FIELD_PREP(SDXI_MMIO_CTL2_MAX_CXT, sdxi->max_cxtid);
+	ctl2 |= FIELD_PREP(SDXI_MMIO_CTL2_MAX_BUFFER,
+			   sdxi->max_buffer);
+	ctl2 |= FIELD_PREP(SDXI_MMIO_CTL2_MAX_AKEY_SZ,
+			   FIELD_GET(SDXI_MMIO_CAP1_MAX_AKEY_SZ, cap1));
+	ctl2 |= FIELD_PREP(SDXI_MMIO_CTL2_OPB_000_AVL,
+			   FIELD_GET(SDXI_MMIO_CAP1_OPB_000_CAP, cap1));
+	sdxi_write64(sdxi, SDXI_MMIO_CTL2, ctl2);
+
+	return 0;
+}
+
 int sdxi_register(struct device *dev, const struct sdxi_bus_ops *ops)
 {
 	struct sdxi_dev *sdxi;
@@ -292,7 +333,11 @@ int sdxi_register(struct device *dev, const struct sdxi_bus_ops *ops)
 	if (err)
 		return err;
 
-	return sdxi_fn_prepare(sdxi);
+	err = sdxi_fn_prepare(sdxi);
+	if (err)
+		return err;
+
+	return sdxi_fn_activate(sdxi);
 }
 EXPORT_SYMBOL_NS_GPL(sdxi_register, "SDXI");
 
diff --git a/drivers/dma/sdxi/mmio.h b/drivers/dma/sdxi/mmio.h
index a9c1fbb404d5..7c416de9f874 100644
--- a/drivers/dma/sdxi/mmio.h
+++ b/drivers/dma/sdxi/mmio.h
@@ -16,7 +16,9 @@ enum sdxi_reg {
 	SDXI_MMIO_CTL0       = 0x00000,
 	SDXI_MMIO_CTL2       = 0x00010,
 	SDXI_MMIO_STS0       = 0x00100,
+	SDXI_MMIO_CAP0       = 0x00200,
 	SDXI_MMIO_CAP1       = 0x00208,
+	SDXI_MMIO_VERSION    = 0x00210,
 
 	/* SDXI 1.0 9.2 Context and RKey Table Registers */
 	SDXI_MMIO_CXT_L2     = 0x10000,
@@ -48,6 +50,11 @@ enum sdxi_reg {
 /* SDXI 1.0 Table 9-5: MMIO_STS0 */
 #define SDXI_MMIO_STS0_FN_GSV GENMASK_ULL(2, 0)
 
+/* SDXI 1.0 Table 9-6: MMIO_CAP0 */
+#define SDXI_MMIO_CAP0_SFUNC          GENMASK_ULL(15, 0)
+#define SDXI_MMIO_CAP0_DB_STRIDE      GENMASK_ULL(22, 20)
+#define SDXI_MMIO_CAP0_MAX_DS_RING_SZ GENMASK_ULL(28, 24)
+
 /* SDXI 1.0 Table 9-7: MMIO_CAP1 */
 #define SDXI_MMIO_CAP1_MAX_BUFFER  GENMASK_ULL(3, 0)
 #define SDXI_MMIO_CAP1_RKEY_CAP    BIT_ULL(4)
@@ -55,6 +62,12 @@ enum sdxi_reg {
 #define SDXI_MMIO_CAP1_MAX_AKEY_SZ GENMASK_ULL(15, 12)
 #define SDXI_MMIO_CAP1_MAX_CXT     GENMASK_ULL(31, 16)
 
+#define SDXI_MMIO_CAP1_OPB_000_CAP GENMASK_ULL(63, 32)
+
+/* SDXI 1.0 Table 9-8: MMIO_VERSION */
+#define SDXI_MMIO_VERSION_MINOR GENMASK_ULL(7, 0)
+#define SDXI_MMIO_VERSION_MAJOR GENMASK_ULL(23, 16)
+
 /* SDXI 1.0 Table 9-10: MMIO_RKEY */
 #define SDXI_MMIO_RKEY_EN BIT_ULL(0)
 
diff --git a/drivers/dma/sdxi/sdxi.h b/drivers/dma/sdxi/sdxi.h
index bb0d2165d495..7569b8395ce1 100644
--- a/drivers/dma/sdxi/sdxi.h
+++ b/drivers/dma/sdxi/sdxi.h
@@ -38,6 +38,12 @@ struct sdxi_dev {
 	void __iomem *ctrl_regs;	/* virt addr of ctrl registers */
 	void __iomem *dbs;		/* virt addr of doorbells */
 
+	/* hardware capabilities (from cap0 & cap1) */
+	u32 db_stride;			/* doorbell stride in bytes */
+	u16 max_cxtid;			/* Maximum context ID allowed. */
+	u8 max_buffer;			/* Configured log2 buffer size minus 21. */
+	u32 op_grp_cap;			/* supported operation group cap */
+
 	bool has_mailbox;
 
 	const struct sdxi_bus_ops *bus_ops;

-- 
2.56.0

[PATCH v4 06/25] dmaengine: sdxi: Configure context tables

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

SDXI uses a two-level table hierarchy to track contexts. There is a
single level 2 table per function which enumerates up to 512 level 1
tables. Each level 1 table enumerates up to 128 contexts.

Allocate and install the L2 table and a single L1 table, enough for
context IDs 0-127 (i.e. the admin context with reserved id 0, plus 127
client contexts). For now, to avoid dynamic management of additional
L1 tables, cap ctl2.max_cxt to 127.

Since the table allocations are devres-managed, there is no
corresponding cleanup code required.

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/device.c | 39 ++++++++++++++++++++++++++++++++++--
 drivers/dma/sdxi/hw.h     | 50 +++++++++++++++++++++++++++++++++++++++++++++++
 drivers/dma/sdxi/mmio.h   |  3 +++
 drivers/dma/sdxi/sdxi.h   |  5 +++++
 4 files changed, 95 insertions(+), 2 deletions(-)

diff --git a/drivers/dma/sdxi/device.c b/drivers/dma/sdxi/device.c
index c03af5833414..baf55f6b6826 100644
--- a/drivers/dma/sdxi/device.c
+++ b/drivers/dma/sdxi/device.c
@@ -13,11 +13,13 @@
 #include <linux/iopoll.h>
 #include <linux/jiffies.h>
 #include <linux/limits.h>
+#include <linux/log2.h>
 #include <linux/minmax.h>
 #include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/time.h>
 
+#include "hw.h"
 #include "mmio.h"
 #include "sdxi.h"
 
@@ -284,7 +286,8 @@ static int sdxi_fn_prepare(struct sdxi_dev *sdxi)
  */
 static int sdxi_fn_activate(struct sdxi_dev *sdxi)
 {
-	u64 version, cap0, cap1, ctl2;
+	u64 version, cap0, cap1, ctl2, cxt_l2, lv01_ptr;
+	struct sdxi_cxt_L2_ent *L2_ent;
 
 	version = sdxi_read64(sdxi, SDXI_MMIO_VERSION);
 	dev_info(sdxi->dev, "SDXI %llu.%llu device found\n",
@@ -301,7 +304,13 @@ static int sdxi_fn_activate(struct sdxi_dev *sdxi)
 	sdxi->max_buffer = FIELD_GET(SDXI_MMIO_CAP1_MAX_BUFFER, cap1);
 	dma_set_max_seg_size(sdxi->dev,
 			     min((u64)SZ_2M << sdxi->max_buffer, U32_MAX));
-	sdxi->max_cxtid = FIELD_GET(SDXI_MMIO_CAP1_MAX_CXT, cap1);
+
+	/*
+	 * Constrain the number of client contexts supported by the
+	 * driver to what fits in a single L1 table.
+	 */
+	sdxi->max_cxtid = min(SDXI_L1_TABLE_ENTRIES - 1,
+			      FIELD_GET(SDXI_MMIO_CAP1_MAX_CXT, cap1));
 
 	/* Apply our configuration. */
 	ctl2 = FIELD_PREP(SDXI_MMIO_CTL2_MAX_CXT, sdxi->max_cxtid);
@@ -313,6 +322,32 @@ static int sdxi_fn_activate(struct sdxi_dev *sdxi)
 			   FIELD_GET(SDXI_MMIO_CAP1_OPB_000_CAP, cap1));
 	sdxi_write64(sdxi, SDXI_MMIO_CTL2, ctl2);
 
+	/* SDXI 1.0 4.1.8.2 Context Level 2 Table Setup */
+	sdxi->L2_table = dmam_alloc_coherent(sdxi->dev,
+					     sizeof(*sdxi->L2_table),
+					     &sdxi->L2_dma, GFP_KERNEL);
+	if (!sdxi->L2_table)
+		return -ENOMEM;
+
+	cxt_l2 = FIELD_PREP(SDXI_MMIO_CXT_L2_PTR, sdxi->L2_dma >> ilog2(SZ_4K));
+	sdxi_write64(sdxi, SDXI_MMIO_CXT_L2, cxt_l2);
+
+	/* SDXI 1.0 4.1.8.3 Context Level 1 Table Setup */
+	sdxi->L1_table = dmam_alloc_coherent(sdxi->dev,
+					     sizeof(*sdxi->L1_table),
+					     &sdxi->L1_dma, GFP_KERNEL);
+	if (!sdxi->L1_table)
+		return -ENOMEM;
+	/*
+	 * SDXI 1.0 4.1.8.3.c: Initialize the Context level 2 table to
+	 * point to the Context Level 1 [table].
+	 */
+	L2_ent = &sdxi->L2_table->entry[0];
+	lv01_ptr = FIELD_PREP(SDXI_CXT_L2_ENT_VL, 1) |
+		   FIELD_PREP(SDXI_CXT_L2_ENT_LV01_PTR,
+			      sdxi->L1_dma >> ilog2(SZ_4K));
+	L2_ent->lv01_ptr = cpu_to_le64(lv01_ptr);
+
 	return 0;
 }
 
diff --git a/drivers/dma/sdxi/hw.h b/drivers/dma/sdxi/hw.h
new file mode 100644
index 000000000000..0551c2b52980
--- /dev/null
+++ b/drivers/dma/sdxi/hw.h
@@ -0,0 +1,50 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright Advanced Micro Devices, Inc. */
+
+/*
+ * Hardware layouts and constants follow SDXI specification names and
+ * definition order. Reserved fields and bits must be zero unless specified
+ * otherwise.
+ */
+
+#ifndef DMA_SDXI_HW_H
+#define DMA_SDXI_HW_H
+
+#include <linux/bits.h>
+#include <linux/build_bug.h>
+#include <linux/types.h>
+#include <asm/byteorder.h>
+
+/* SDXI 1.0 Table 3-2: Context Level 2 Table Entry (CXT_L2_ENT) */
+struct sdxi_cxt_L2_ent {
+	__le64 lv01_ptr;
+#define SDXI_CXT_L2_ENT_VL       BIT_ULL(0)
+#define SDXI_CXT_L2_ENT_LV01_PTR GENMASK_ULL(63, 12)
+} __packed __aligned(8);
+static_assert(sizeof(struct sdxi_cxt_L2_ent) == 8);
+
+/* SDXI 1.0 3.2.1 Context Level 2 Table */
+#define SDXI_L2_TABLE_ENTRIES 512
+struct sdxi_cxt_L2_table {
+	struct sdxi_cxt_L2_ent entry[SDXI_L2_TABLE_ENTRIES];
+};
+static_assert(sizeof(struct sdxi_cxt_L2_table) == 4096);
+
+/* SDXI 1.0 Table 3-3: Context Level 1 Table Entry (CXT_L1_ENT) */
+struct sdxi_cxt_L1_ent {
+	__le64 cxt_ctl_ptr;
+	__le64 akey_ptr;
+	__le32 misc0;
+	__le32 opb_000_enb;
+	__u8 rsvd_0[8];
+} __packed __aligned(32);
+static_assert(sizeof(struct sdxi_cxt_L1_ent) == 32);
+
+/* SDXI 1.0 3.2.2 Context Level 1 Table */
+#define SDXI_L1_TABLE_ENTRIES 128
+struct sdxi_cxt_L1_table {
+	struct sdxi_cxt_L1_ent entry[SDXI_L1_TABLE_ENTRIES];
+};
+static_assert(sizeof(struct sdxi_cxt_L1_table) == 4096);
+
+#endif /* DMA_SDXI_HW_H */
diff --git a/drivers/dma/sdxi/mmio.h b/drivers/dma/sdxi/mmio.h
index 7c416de9f874..b040d2283fb2 100644
--- a/drivers/dma/sdxi/mmio.h
+++ b/drivers/dma/sdxi/mmio.h
@@ -68,6 +68,9 @@ enum sdxi_reg {
 #define SDXI_MMIO_VERSION_MINOR GENMASK_ULL(7, 0)
 #define SDXI_MMIO_VERSION_MAJOR GENMASK_ULL(23, 16)
 
+/* SDXI 1.0 Table 9-9: MMIO_CXT_L2 */
+#define SDXI_MMIO_CXT_L2_PTR GENMASK_ULL(63, 12)
+
 /* SDXI 1.0 Table 9-10: MMIO_RKEY */
 #define SDXI_MMIO_RKEY_EN BIT_ULL(0)
 
diff --git a/drivers/dma/sdxi/sdxi.h b/drivers/dma/sdxi/sdxi.h
index 7569b8395ce1..856a3838eec3 100644
--- a/drivers/dma/sdxi/sdxi.h
+++ b/drivers/dma/sdxi/sdxi.h
@@ -44,6 +44,11 @@ struct sdxi_dev {
 	u8 max_buffer;			/* Configured log2 buffer size minus 21. */
 	u32 op_grp_cap;			/* supported operation group cap */
 
+	struct sdxi_cxt_L2_table *L2_table;
+	dma_addr_t L2_dma;
+	struct sdxi_cxt_L1_table *L1_table;
+	dma_addr_t L1_dma;
+
 	bool has_mailbox;
 
 	const struct sdxi_bus_ops *bus_ops;

-- 
2.56.0

[PATCH v4 07/25] dmaengine: sdxi: Allocate context control DMA pool

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Each SDXI context has a control block (CXT_CTL) that identifies its
descriptor ring, context status block, and write index.

Define the CXT_CTL layout and create a DMA pool for them during
function initialization.

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/device.c | 21 ++++++++++++++++++++-
 drivers/dma/sdxi/hw.h     | 11 +++++++++++
 drivers/dma/sdxi/sdxi.h   |  2 ++
 3 files changed, 33 insertions(+), 1 deletion(-)

diff --git a/drivers/dma/sdxi/device.c b/drivers/dma/sdxi/device.c
index baf55f6b6826..acc955b088ba 100644
--- a/drivers/dma/sdxi/device.c
+++ b/drivers/dma/sdxi/device.c
@@ -9,6 +9,7 @@
 #include <linux/delay.h>
 #include <linux/device.h>
 #include <linux/dma-mapping.h>
+#include <linux/dmapool.h>
 #include <linux/export.h>
 #include <linux/iopoll.h>
 #include <linux/jiffies.h>
@@ -351,6 +352,24 @@ static int sdxi_fn_activate(struct sdxi_dev *sdxi)
 	return 0;
 }
 
+static int sdxi_device_init(struct sdxi_dev *sdxi)
+{
+	struct device *dev = sdxi->dev;
+	size_t size, align;
+	int err;
+
+	size = align = sizeof(struct sdxi_cxt_ctl);
+	sdxi->cxt_ctl_pool = dmam_pool_create("CXT_CTL", dev, size, align, 0);
+	if (!sdxi->cxt_ctl_pool)
+		return -ENOMEM;
+
+	err = sdxi_fn_activate(sdxi);
+	if (err)
+		return err;
+
+	return 0;
+}
+
 int sdxi_register(struct device *dev, const struct sdxi_bus_ops *ops)
 {
 	struct sdxi_dev *sdxi;
@@ -372,7 +391,7 @@ int sdxi_register(struct device *dev, const struct sdxi_bus_ops *ops)
 	if (err)
 		return err;
 
-	return sdxi_fn_activate(sdxi);
+	return sdxi_device_init(sdxi);
 }
 EXPORT_SYMBOL_NS_GPL(sdxi_register, "SDXI");
 
diff --git a/drivers/dma/sdxi/hw.h b/drivers/dma/sdxi/hw.h
index 0551c2b52980..82c6b4a20cde 100644
--- a/drivers/dma/sdxi/hw.h
+++ b/drivers/dma/sdxi/hw.h
@@ -47,4 +47,15 @@ struct sdxi_cxt_L1_table {
 };
 static_assert(sizeof(struct sdxi_cxt_L1_table) == 4096);
 
+/* SDXI 1.0 Table 3-4: Context Control (CXT_CTL) */
+struct sdxi_cxt_ctl {
+	__le64 ds_ring_ptr;
+	__le32 ds_ring_sz;
+	__u8 rsvd_0[4];
+	__le64 cxt_sts_ptr;
+	__le64 write_index_ptr;
+	__u8 rsvd_1[32];
+} __packed __aligned(64);
+static_assert(sizeof(struct sdxi_cxt_ctl) == 64);
+
 #endif /* DMA_SDXI_HW_H */
diff --git a/drivers/dma/sdxi/sdxi.h b/drivers/dma/sdxi/sdxi.h
index 856a3838eec3..356a47ed39da 100644
--- a/drivers/dma/sdxi/sdxi.h
+++ b/drivers/dma/sdxi/sdxi.h
@@ -49,6 +49,8 @@ struct sdxi_dev {
 	struct sdxi_cxt_L1_table *L1_table;
 	dma_addr_t L1_dma;
 
+	struct dma_pool *cxt_ctl_pool;
+
 	bool has_mailbox;
 
 	const struct sdxi_bus_ops *bus_ops;

-- 
2.56.0

[PATCH v4 08/25] dmaengine: sdxi: Introduce submission queue

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Add the sdxi_sq (submission queue) type and APIs that act on it. The
sq contains three control structures shared with hardware: descriptor
ring, write index, and context status block (which contains the read
index). Define the hardware context-status and generic descriptor
layouts, including assertions for their sizes.

The descriptor ring requires only 64-byte alignment, so it's practical
to implement it as a flexible array member. The write index and context
status are small enough to place in the struct as well, so we have a
single DMA coherent allocation to manage.

Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/hw.h         | 25 ++++++++++++++++
 drivers/dma/sdxi/submission.h | 66 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 91 insertions(+)

diff --git a/drivers/dma/sdxi/hw.h b/drivers/dma/sdxi/hw.h
index 82c6b4a20cde..82a9c20d8bcf 100644
--- a/drivers/dma/sdxi/hw.h
+++ b/drivers/dma/sdxi/hw.h
@@ -12,6 +12,7 @@
 
 #include <linux/bits.h>
 #include <linux/build_bug.h>
+#include <linux/stddef.h>
 #include <linux/types.h>
 #include <asm/byteorder.h>
 
@@ -58,4 +59,28 @@ struct sdxi_cxt_ctl {
 } __packed __aligned(64);
 static_assert(sizeof(struct sdxi_cxt_ctl) == 64);
 
+/* SDXI 1.0 Table 3-5: Context Status (CXT_STS) */
+struct sdxi_cxt_sts {
+	__u8 state;
+	__u8 misc0;
+	__u8 rsvd_0[6];
+	__le64 read_index;
+} __packed __aligned(16);
+static_assert(sizeof(struct sdxi_cxt_sts) == 16);
+
+struct sdxi_desc {
+	union {
+		/*
+		 * SDXI 1.0 Table 6-3: DSC_GENERIC SDXI Descriptor
+		 * Common Header and Footer Format
+		 */
+		struct_group_tagged(sdxi_dsc_generic, generic,
+			__le32 opcode;
+			__u8 operation[52];
+			__le64 csb_ptr;
+		);
+	};
+} __packed __aligned(64);
+static_assert(sizeof(struct sdxi_desc) == 64);
+
 #endif /* DMA_SDXI_HW_H */
diff --git a/drivers/dma/sdxi/submission.h b/drivers/dma/sdxi/submission.h
new file mode 100644
index 000000000000..d60897a373c9
--- /dev/null
+++ b/drivers/dma/sdxi/submission.h
@@ -0,0 +1,66 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright Advanced Micro Devices, Inc. */
+#ifndef DMA_SDXI_SUBMISSION_H
+#define DMA_SDXI_SUBMISSION_H
+
+#include <linux/cache.h>
+#include <linux/compiler_attributes.h>
+#include <linux/dma-mapping.h>
+#include <linux/overflow.h>
+#include <linux/types.h>
+
+#include "hw.h"
+#include "sdxi.h"
+
+/*
+ * Submission queue: one DMA-coherent allocation holding the
+ * descriptor ring, write index, and context status (which includes
+ * the read index). cxt_sts gets its own cache line to avoid sharing
+ * with write_index.
+ */
+struct sdxi_sq {
+	dma_addr_t handle;
+	void __iomem *doorbell;
+	__le64 write_index;
+	u32 count;
+	struct sdxi_cxt_sts cxt_sts ____cacheline_aligned;
+	struct sdxi_desc ring[] __counted_by(count);
+};
+
+static inline struct sdxi_sq *sdxi_sq_alloc(struct sdxi_dev *sdxi, unsigned int count)
+{
+	dma_addr_t handle;
+	struct sdxi_sq *sq;
+	size_t sz = struct_size(sq, ring, count);
+
+	sq = dma_alloc_coherent(sdxi->dev, sz, &handle, GFP_KERNEL);
+	if (!sq)
+		return NULL;
+
+	sq->count = count;
+	sq->handle = handle;
+	return sq;
+}
+
+static inline void sdxi_sq_free(struct sdxi_dev *sdxi, struct sdxi_sq *sq)
+{
+	dma_free_coherent(sdxi->dev, struct_size(sq, ring, sq->count), sq,
+			  sq->handle);
+}
+
+static inline dma_addr_t sdxi_sq_ring_dma(const struct sdxi_sq *sq)
+{
+	return sq->handle + struct_offset(sq, ring);
+}
+
+static inline dma_addr_t sdxi_sq_write_index_dma(const struct sdxi_sq *sq)
+{
+	return sq->handle + struct_offset(sq, write_index);
+}
+
+static inline dma_addr_t sdxi_sq_cxt_sts_dma(const struct sdxi_sq *sq)
+{
+	return sq->handle + struct_offset(sq, cxt_sts);
+}
+
+#endif /* DMA_SDXI_SUBMISSION_H */

-- 
2.56.0

[PATCH v4 09/25] dmaengine: sdxi: Allocate administrative context

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Allocate the administrative context’s control structures and register
device-managed cleanup.

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/Makefile  |   4 +-
 drivers/dma/sdxi/context.c | 101 +++++++++++++++++++++++++++++++++++++++++++++
 drivers/dma/sdxi/context.h |  36 ++++++++++++++++
 drivers/dma/sdxi/device.c  |  12 ++++++
 drivers/dma/sdxi/hw.h      |  27 ++++++++++++
 drivers/dma/sdxi/sdxi.h    |   2 +
 6 files changed, 181 insertions(+), 1 deletion(-)

diff --git a/drivers/dma/sdxi/Makefile b/drivers/dma/sdxi/Makefile
index 0006edf74d86..cdf8a455077b 100644
--- a/drivers/dma/sdxi/Makefile
+++ b/drivers/dma/sdxi/Makefile
@@ -1,6 +1,8 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_SDXI_CORE) += sdxi-core.o
-sdxi-core-y := device.o
+sdxi-core-y := \
+	context.o     \
+	device.o
 
 obj-$(CONFIG_SDXI_PCI) += sdxi-pci.o
 sdxi-pci-y := pci.o
diff --git a/drivers/dma/sdxi/context.c b/drivers/dma/sdxi/context.c
new file mode 100644
index 000000000000..4132618ae17a
--- /dev/null
+++ b/drivers/dma/sdxi/context.c
@@ -0,0 +1,101 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * SDXI context management
+ *
+ * Copyright Advanced Micro Devices, Inc.
+ */
+
+#define pr_fmt(fmt)     "SDXI: " fmt
+
+#include <linux/bug.h>
+#include <linux/cleanup.h>
+#include <linux/device/devres.h>
+#include <linux/dma-mapping.h>
+#include <linux/dmapool.h>
+#include <linux/errno.h>
+#include <linux/slab.h>
+#include <linux/types.h>
+
+#include "context.h"
+#include "sdxi.h"
+#include "submission.h"
+
+#define DEFAULT_DESC_RING_ENTRIES 1024
+
+enum {
+	/*
+	 * The admin context always has ID 0. See SDXI 1.0 3.5
+	 * Administrative Context (Context 0).
+	 */
+	SDXI_ADMIN_CXT_ID = 0,
+};
+
+/*
+ * Free context and its resources. @cxt may be partially allocated but
+ * must have ->sdxi set.
+ */
+static void sdxi_free_cxt(struct sdxi_cxt *cxt)
+{
+	struct sdxi_dev *sdxi = cxt->sdxi;
+
+	if (cxt->cxt_ctl)
+		dma_pool_free(sdxi->cxt_ctl_pool, cxt->cxt_ctl,
+			      cxt->cxt_ctl_dma);
+	if (cxt->akey_table)
+		dma_free_coherent(sdxi->dev, sizeof(*cxt->akey_table),
+				  cxt->akey_table, cxt->akey_table_dma);
+	if (cxt->sq)
+		sdxi_sq_free(sdxi, cxt->sq);
+	kfree(cxt);
+}
+
+DEFINE_FREE(sdxi_cxt, struct sdxi_cxt *, if (_T) sdxi_free_cxt(_T))
+
+/* Allocate a context and its control structure hierarchy in memory. */
+static struct sdxi_cxt *sdxi_alloc_cxt(struct sdxi_dev *sdxi)
+{
+	struct device *dev = sdxi->dev;
+	struct sdxi_cxt *cxt __free(sdxi_cxt) = kzalloc(sizeof(*cxt), GFP_KERNEL);
+
+	if (!cxt)
+		return NULL;
+
+	cxt->sdxi = sdxi;
+
+	cxt->sq = sdxi_sq_alloc(sdxi, DEFAULT_DESC_RING_ENTRIES);
+	if (!cxt->sq)
+		return NULL;
+
+	cxt->akey_table = dma_alloc_coherent(dev, sizeof(*cxt->akey_table),
+					     &cxt->akey_table_dma, GFP_KERNEL);
+	if (!cxt->akey_table)
+		return NULL;
+
+	cxt->cxt_ctl = dma_pool_zalloc(sdxi->cxt_ctl_pool, GFP_KERNEL,
+				       &cxt->cxt_ctl_dma);
+	if (!cxt->cxt_ctl)
+		return NULL;
+
+	return_ptr(cxt);
+}
+
+static void free_admin_cxt(void *ptr)
+{
+	struct sdxi_dev *sdxi = ptr;
+
+	sdxi_free_cxt(sdxi->admin_cxt);
+}
+
+int sdxi_admin_cxt_init(struct sdxi_dev *sdxi)
+{
+	struct sdxi_cxt *cxt __free(sdxi_cxt) = sdxi_alloc_cxt(sdxi);
+	if (!cxt)
+		return -ENOMEM;
+
+	cxt->id = SDXI_ADMIN_CXT_ID;
+	cxt->sq->doorbell = sdxi->dbs + cxt->id * sdxi->db_stride;
+
+	sdxi->admin_cxt = no_free_ptr(cxt);
+
+	return devm_add_action_or_reset(sdxi->dev, free_admin_cxt, sdxi);
+}
diff --git a/drivers/dma/sdxi/context.h b/drivers/dma/sdxi/context.h
new file mode 100644
index 000000000000..873ad49c3c86
--- /dev/null
+++ b/drivers/dma/sdxi/context.h
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright Advanced Micro Devices, Inc.
+ */
+
+#ifndef DMA_SDXI_CONTEXT_H
+#define DMA_SDXI_CONTEXT_H
+
+#include <linux/dma-mapping.h>
+#include <linux/types.h>
+
+#include "hw.h"
+#include "sdxi.h"
+#include "submission.h"
+
+/* Use a 4 KiB AKey table, the minimum architectural size. */
+struct sdxi_akey_table {
+	struct sdxi_akey_ent entry[SZ_4K / sizeof(struct sdxi_akey_ent)];
+};
+
+struct sdxi_cxt {
+	struct sdxi_dev *sdxi;
+	u16 id;
+
+	struct sdxi_cxt_ctl *cxt_ctl;
+	dma_addr_t cxt_ctl_dma;
+
+	struct sdxi_akey_table *akey_table;
+	dma_addr_t akey_table_dma;
+
+	struct sdxi_sq *sq;
+};
+
+int sdxi_admin_cxt_init(struct sdxi_dev *sdxi);
+
+#endif /* DMA_SDXI_CONTEXT_H */
diff --git a/drivers/dma/sdxi/device.c b/drivers/dma/sdxi/device.c
index acc955b088ba..f9787d7190d4 100644
--- a/drivers/dma/sdxi/device.c
+++ b/drivers/dma/sdxi/device.c
@@ -20,6 +20,7 @@
 #include <linux/slab.h>
 #include <linux/time.h>
 
+#include "context.h"
 #include "hw.h"
 #include "mmio.h"
 #include "sdxi.h"
@@ -289,6 +290,7 @@ static int sdxi_fn_activate(struct sdxi_dev *sdxi)
 {
 	u64 version, cap0, cap1, ctl2, cxt_l2, lv01_ptr;
 	struct sdxi_cxt_L2_ent *L2_ent;
+	int err;
 
 	version = sdxi_read64(sdxi, SDXI_MMIO_VERSION);
 	dev_info(sdxi->dev, "SDXI %llu.%llu device found\n",
@@ -349,6 +351,16 @@ static int sdxi_fn_activate(struct sdxi_dev *sdxi)
 			      sdxi->L1_dma >> ilog2(SZ_4K));
 	L2_ent->lv01_ptr = cpu_to_le64(lv01_ptr);
 
+	/*
+	 * SDXI 1.0 4.1.8.4 Administrative Context
+	 *
+	 * The admin context will not consume descriptors until we
+	 * write its doorbell later.
+	 */
+	err = sdxi_admin_cxt_init(sdxi);
+	if (err)
+		return err;
+
 	return 0;
 }
 
diff --git a/drivers/dma/sdxi/hw.h b/drivers/dma/sdxi/hw.h
index 82a9c20d8bcf..d1917dad243f 100644
--- a/drivers/dma/sdxi/hw.h
+++ b/drivers/dma/sdxi/hw.h
@@ -62,12 +62,39 @@ static_assert(sizeof(struct sdxi_cxt_ctl) == 64);
 /* SDXI 1.0 Table 3-5: Context Status (CXT_STS) */
 struct sdxi_cxt_sts {
 	__u8 state;
+#define SDXI_CXT_STS_STATE GENMASK(3, 0)
 	__u8 misc0;
 	__u8 rsvd_0[6];
 	__le64 read_index;
 } __packed __aligned(16);
 static_assert(sizeof(struct sdxi_cxt_sts) == 16);
 
+/* SDXI 1.0 Table 3-6: CXT_STS.state Encoding */
+/* Valid values for FIELD_GET(SDXI_CXT_STS_STATE, sdxi_cxt_sts.state). */
+enum cxt_sts_state {
+	CXTV_STOP_SW  = 0x0,
+	CXTV_RUN      = 0x1,
+	CXTV_STOPG_SW = 0x2,
+	CXTV_STOP_FN  = 0x4,
+	CXTV_STOPG_FN = 0x6,
+	CXTV_ERR_FN   = 0xf,
+};
+
+/* SDXI 1.0 Table 3-7: AKey Table Entry (AKEY_ENT) */
+struct sdxi_akey_ent {
+	__le16 intr_num;
+#define SDXI_AKEY_ENT_VL BIT(0)
+#define SDXI_AKEY_ENT_IV BIT(1)
+#define SDXI_AKEY_ENT_INTR_NUM GENMASK(14, 4)
+	__le16 tgt_sfunc;
+	__le32 pasid;
+	__le16 stag;
+	__u8   rsvd_0[2];
+	__le16 rkey;
+	__u8   rsvd_1[2];
+} __packed __aligned(16);
+static_assert(sizeof(struct sdxi_akey_ent) == 16);
+
 struct sdxi_desc {
 	union {
 		/*
diff --git a/drivers/dma/sdxi/sdxi.h b/drivers/dma/sdxi/sdxi.h
index 356a47ed39da..b7d754132268 100644
--- a/drivers/dma/sdxi/sdxi.h
+++ b/drivers/dma/sdxi/sdxi.h
@@ -51,6 +51,8 @@ struct sdxi_dev {
 
 	struct dma_pool *cxt_ctl_pool;
 
+	struct sdxi_cxt *admin_cxt;
+
 	bool has_mailbox;
 
 	const struct sdxi_bus_ops *bus_ops;

-- 
2.56.0

[PATCH v4 10/25] dmaengine: sdxi: Install administrative context

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Serialize the context control block, akey table, and L1 entry for the
admin context, making its descriptor ring, write index, and context
status block visible to the SDXI implementation once it is activated.

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/context.c | 149 +++++++++++++++++++++++++++++++++++++++++++++
 drivers/dma/sdxi/context.h |   7 +++
 drivers/dma/sdxi/hw.h      |  15 +++++
 drivers/dma/sdxi/sdxi.h    |   9 +++
 4 files changed, 180 insertions(+)

diff --git a/drivers/dma/sdxi/context.c b/drivers/dma/sdxi/context.c
index 4132618ae17a..ebce30dc462a 100644
--- a/drivers/dma/sdxi/context.c
+++ b/drivers/dma/sdxi/context.c
@@ -7,16 +7,22 @@
 
 #define pr_fmt(fmt)     "SDXI: " fmt
 
+#include <linux/align.h>
+#include <linux/bitfield.h>
 #include <linux/bug.h>
 #include <linux/cleanup.h>
 #include <linux/device/devres.h>
 #include <linux/dma-mapping.h>
 #include <linux/dmapool.h>
 #include <linux/errno.h>
+#include <linux/iommu.h>
 #include <linux/slab.h>
 #include <linux/types.h>
+#include <asm/barrier.h>
+#include <asm/rwonce.h>
 
 #include "context.h"
+#include "hw.h"
 #include "sdxi.h"
 #include "submission.h"
 
@@ -79,6 +85,139 @@ static struct sdxi_cxt *sdxi_alloc_cxt(struct sdxi_dev *sdxi)
 	return_ptr(cxt);
 }
 
+struct sdxi_cxt_ctl_cfg {
+	dma_addr_t ds_ring_ptr;
+	dma_addr_t cxt_sts_ptr;
+	dma_addr_t write_index_ptr;
+	u32 ds_ring_sz;
+	u8 qos;
+	u8 csa;
+	bool se;
+};
+
+static int configure_cxt_ctl(struct sdxi_cxt_ctl *ctl, const struct sdxi_cxt_ctl_cfg *cfg)
+{
+	u64 ds_ring_ptr, cxt_sts_ptr, write_index_ptr;
+
+	write_index_ptr = FIELD_PREP(SDXI_CXT_CTL_WRITE_INDEX_PTR,
+				     cfg->write_index_ptr >> WRT_INDEX_PTR_SHIFT);
+	cxt_sts_ptr = FIELD_PREP(SDXI_CXT_CTL_CXT_STS_PTR,
+				 cfg->cxt_sts_ptr >> CXT_STATUS_PTR_SHIFT);
+
+	*ctl = (typeof(*ctl)) {
+		.ds_ring_sz      = cpu_to_le32(cfg->ds_ring_sz),
+		.cxt_sts_ptr     = cpu_to_le64(cxt_sts_ptr),
+		.write_index_ptr = cpu_to_le64(write_index_ptr),
+	};
+
+	ds_ring_ptr = FIELD_PREP(SDXI_CXT_CTL_VL, 1) |
+		      FIELD_PREP(SDXI_CXT_CTL_QOS, cfg->qos) |
+		      FIELD_PREP(SDXI_CXT_CTL_SE, cfg->se) |
+		      FIELD_PREP(SDXI_CXT_CTL_CSA, cfg->csa) |
+		      FIELD_PREP(SDXI_CXT_CTL_DS_RING_PTR,
+				 cfg->ds_ring_ptr >> DESC_RING_BASE_PTR_SHIFT);
+	/* Ensure other fields are visible before hw sees vl=1. */
+	dma_wmb();
+	WRITE_ONCE(ctl->ds_ring_ptr, cpu_to_le64(ds_ring_ptr));
+
+	return 0;
+}
+
+/*
+ * Logical representation of CXT_L1_ENT subfields.
+ */
+struct sdxi_cxt_L1_cfg {
+	dma_addr_t cxt_ctl_ptr;
+	dma_addr_t akey_ptr;
+	u32 cxt_pasid;
+	u32 opb_000_enb;
+	u16 max_buffer;
+	u8 akey_sz;
+	bool ka;
+	bool pv;
+};
+
+static int configure_L1_entry(struct sdxi_cxt_L1_ent *ent,
+			      const struct sdxi_cxt_L1_cfg *cfg)
+{
+	u64 cxt_ctl_ptr, akey_ptr;
+	u32 misc0;
+
+	if (WARN_ON_ONCE(!IS_ALIGNED(cfg->cxt_ctl_ptr, SZ_64)))
+		return -EFAULT;
+	if (WARN_ON_ONCE(!IS_ALIGNED(cfg->akey_ptr, SZ_4K)))
+		return -EFAULT;
+
+	akey_ptr = FIELD_PREP(SDXI_CXT_L1_ENT_AKEY_SZ, cfg->akey_sz) |
+		   FIELD_PREP(SDXI_CXT_L1_ENT_AKEY_PTR,
+			      cfg->akey_ptr >> L1_CXT_AKEY_PTR_SHIFT);
+
+	misc0 = FIELD_PREP(SDXI_CXT_L1_ENT_PASID, cfg->cxt_pasid) |
+		FIELD_PREP(SDXI_CXT_L1_ENT_MAX_BUFFER, cfg->max_buffer);
+
+	*ent = (typeof(*ent)) {
+		.akey_ptr    = cpu_to_le64(akey_ptr),
+		.misc0       = cpu_to_le32(misc0),
+		.opb_000_enb = cpu_to_le32(cfg->opb_000_enb),
+	};
+
+	cxt_ctl_ptr = FIELD_PREP(SDXI_CXT_L1_ENT_VL, 1) |
+		      FIELD_PREP(SDXI_CXT_L1_ENT_KA, cfg->ka) |
+		      FIELD_PREP(SDXI_CXT_L1_ENT_PV, cfg->pv) |
+		      FIELD_PREP(SDXI_CXT_L1_ENT_CXT_CTL_PTR,
+				 cfg->cxt_ctl_ptr >> L1_CXT_CTRL_PTR_SHIFT);
+	/* Ensure other fields are visible before hw sees vl=1. */
+	dma_wmb();
+	WRITE_ONCE(ent->cxt_ctl_ptr, cpu_to_le64(cxt_ctl_ptr));
+
+	return 0;
+}
+
+/* Publish the context control block and L1 entry to hardware. */
+static int sdxi_publish_cxt(const struct sdxi_cxt *cxt)
+{
+	struct sdxi_cxt_ctl_cfg ctl_cfg;
+	struct sdxi_cxt_L1_cfg L1_cfg;
+	struct sdxi_cxt_L1_ent *ent;
+	u8 l1_idx;
+	int err;
+
+	if (WARN_ONCE(cxt->id > cxt->sdxi->max_cxtid,
+		      "can't install cxt with id %u (limit %u)",
+		      cxt->id, cxt->sdxi->max_cxtid))
+		return -EINVAL;
+
+	ctl_cfg = (typeof(ctl_cfg)) {
+		.se              = 1,
+		.csa             = 1,
+		.ds_ring_ptr     = sdxi_sq_ring_dma(cxt->sq),
+		.ds_ring_sz      = cxt->sq->count,
+		.cxt_sts_ptr     = sdxi_sq_cxt_sts_dma(cxt->sq),
+		.write_index_ptr = sdxi_sq_write_index_dma(cxt->sq),
+	};
+
+	err = configure_cxt_ctl(cxt->cxt_ctl, &ctl_cfg);
+	if (err)
+		return err;
+
+	l1_idx = ID_TO_L1_INDEX(cxt->id);
+
+	ent = &cxt->sdxi->L1_table->entry[l1_idx];
+
+	L1_cfg = (typeof(L1_cfg)) {
+		.ka          = 1,
+		.pv          = 0,
+		.cxt_ctl_ptr = cxt->cxt_ctl_dma,
+		.akey_sz     = akey_table_order(cxt->akey_table),
+		.akey_ptr    = cxt->akey_table_dma,
+		.cxt_pasid   = IOMMU_NO_PASID,
+		.max_buffer  = cxt->sdxi->max_buffer,
+		.opb_000_enb = cxt->sdxi->op_grp_cap,
+	};
+
+	return configure_L1_entry(ent, &L1_cfg);
+}
+
 static void free_admin_cxt(void *ptr)
 {
 	struct sdxi_dev *sdxi = ptr;
@@ -88,13 +227,23 @@ static void free_admin_cxt(void *ptr)
 
 int sdxi_admin_cxt_init(struct sdxi_dev *sdxi)
 {
+	int err;
+	struct sdxi_sq *sq;
+
 	struct sdxi_cxt *cxt __free(sdxi_cxt) = sdxi_alloc_cxt(sdxi);
 	if (!cxt)
 		return -ENOMEM;
 
+	sq = cxt->sq;
+	/* SDXI 1.0 4.1.8.4.b: Set CXT_STS.state to CXTV_RUN. */
+	sq->cxt_sts.state = FIELD_PREP(SDXI_CXT_STS_STATE, CXTV_RUN);
 	cxt->id = SDXI_ADMIN_CXT_ID;
 	cxt->sq->doorbell = sdxi->dbs + cxt->id * sdxi->db_stride;
 
+	err = sdxi_publish_cxt(cxt);
+	if (err)
+		return err;
+
 	sdxi->admin_cxt = no_free_ptr(cxt);
 
 	return devm_add_action_or_reset(sdxi->dev, free_admin_cxt, sdxi);
diff --git a/drivers/dma/sdxi/context.h b/drivers/dma/sdxi/context.h
index 873ad49c3c86..ae1584d00b54 100644
--- a/drivers/dma/sdxi/context.h
+++ b/drivers/dma/sdxi/context.h
@@ -18,6 +18,13 @@ struct sdxi_akey_table {
 	struct sdxi_akey_ent entry[SZ_4K / sizeof(struct sdxi_akey_ent)];
 };
 
+/* For encoding the akey table size in CXT_L1_ENT's akey_sz. */
+static inline u8 akey_table_order(const struct sdxi_akey_table *tbl)
+{
+	static_assert(sizeof(*tbl) == SZ_4K);
+	return 0;
+}
+
 struct sdxi_cxt {
 	struct sdxi_dev *sdxi;
 	u16 id;
diff --git a/drivers/dma/sdxi/hw.h b/drivers/dma/sdxi/hw.h
index d1917dad243f..b333e091c583 100644
--- a/drivers/dma/sdxi/hw.h
+++ b/drivers/dma/sdxi/hw.h
@@ -34,8 +34,16 @@ static_assert(sizeof(struct sdxi_cxt_L2_table) == 4096);
 /* SDXI 1.0 Table 3-3: Context Level 1 Table Entry (CXT_L1_ENT) */
 struct sdxi_cxt_L1_ent {
 	__le64 cxt_ctl_ptr;
+#define SDXI_CXT_L1_ENT_VL             BIT_ULL(0)
+#define SDXI_CXT_L1_ENT_KA             BIT_ULL(1)
+#define SDXI_CXT_L1_ENT_PV             BIT_ULL(2)
+#define SDXI_CXT_L1_ENT_CXT_CTL_PTR    GENMASK_ULL(63, 6)
 	__le64 akey_ptr;
+#define SDXI_CXT_L1_ENT_AKEY_SZ        GENMASK_ULL(3, 0)
+#define SDXI_CXT_L1_ENT_AKEY_PTR       GENMASK_ULL(63, 12)
 	__le32 misc0;
+#define SDXI_CXT_L1_ENT_PASID          GENMASK(19, 0)
+#define SDXI_CXT_L1_ENT_MAX_BUFFER     GENMASK(23, 20)
 	__le32 opb_000_enb;
 	__u8 rsvd_0[8];
 } __packed __aligned(32);
@@ -51,10 +59,17 @@ static_assert(sizeof(struct sdxi_cxt_L1_table) == 4096);
 /* SDXI 1.0 Table 3-4: Context Control (CXT_CTL) */
 struct sdxi_cxt_ctl {
 	__le64 ds_ring_ptr;
+#define SDXI_CXT_CTL_VL             BIT_ULL(0)
+#define SDXI_CXT_CTL_QOS            GENMASK_ULL(3, 2)
+#define SDXI_CXT_CTL_SE             BIT_ULL(4)
+#define SDXI_CXT_CTL_CSA            BIT_ULL(5)
+#define SDXI_CXT_CTL_DS_RING_PTR    GENMASK_ULL(63, 6)
 	__le32 ds_ring_sz;
 	__u8 rsvd_0[4];
 	__le64 cxt_sts_ptr;
+#define SDXI_CXT_CTL_CXT_STS_PTR    GENMASK_ULL(63, 4)
 	__le64 write_index_ptr;
+#define SDXI_CXT_CTL_WRITE_INDEX_PTR GENMASK_ULL(63, 3)
 	__u8 rsvd_1[32];
 } __packed __aligned(64);
 static_assert(sizeof(struct sdxi_cxt_ctl) == 64);
diff --git a/drivers/dma/sdxi/sdxi.h b/drivers/dma/sdxi/sdxi.h
index b7d754132268..525473029987 100644
--- a/drivers/dma/sdxi/sdxi.h
+++ b/drivers/dma/sdxi/sdxi.h
@@ -15,6 +15,15 @@
 
 #include "mmio.h"
 
+#define ID_TO_L1_INDEX(id)	((id) & 0x7F)
+
+#define DESC_RING_BASE_PTR_SHIFT	6
+#define CXT_STATUS_PTR_SHIFT		4
+#define WRT_INDEX_PTR_SHIFT		3
+
+#define L1_CXT_CTRL_PTR_SHIFT		6
+#define L1_CXT_AKEY_PTR_SHIFT		12
+
 struct sdxi_dev;
 
 /**

-- 
2.56.0

[PATCH v4 11/25] dmaengine: sdxi: Start functions on probe, stop on remove

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Activate the function after administrative context setup, polling for
GSV_ACTIVE or GSV_ERROR with a one-second timeout. Stop the function during
PCI removal.

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/device.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++-
 drivers/dma/sdxi/pci.c    |  1 +
 drivers/dma/sdxi/sdxi.h   |  1 +
 3 files changed, 54 insertions(+), 1 deletion(-)

diff --git a/drivers/dma/sdxi/device.c b/drivers/dma/sdxi/device.c
index f9787d7190d4..d5a5b8d43391 100644
--- a/drivers/dma/sdxi/device.c
+++ b/drivers/dma/sdxi/device.c
@@ -212,6 +212,36 @@ static int sdxi_dev_stop(struct sdxi_dev *sdxi)
 	return sdxi->bus_ops->reset ? sdxi->bus_ops->reset(sdxi) : -EIO;
 }
 
+/*
+ * Activate a stopped function. GSRV_ACTIVE takes it through the
+ * transient GSV_INIT state to GSV_ACTIVE or GSV_ERROR. See SDXI 1.0
+ * 4.1.8.9.
+ */
+static int sdxi_dev_start(struct sdxi_dev *sdxi)
+{
+	const unsigned long settled = BIT(SDXI_GSV_ACTIVE) | BIT(SDXI_GSV_ERROR);
+	enum sdxi_fn_gsv gsv = sdxi_dev_gsv(sdxi);
+
+	if (gsv != SDXI_GSV_STOP) {
+		dev_err(sdxi->dev,
+			"start: MMIO_STS0.fn_gsv = %#x (%s) on entry\n",
+			gsv, gsv_str(gsv));
+		return -EBUSY;
+	}
+
+	sdxi_write_fn_gsr(sdxi, SDXI_GSRV_ACTIVE);
+
+	if (sdxi_dev_gsv_poll(sdxi, gsv, BIT(gsv) & settled) ||
+	    gsv != SDXI_GSV_ACTIVE) {
+		dev_err(sdxi->dev,
+			"start: MMIO_STS0.fn_gsv = %#x (%s) after activate\n",
+			gsv, gsv_str(gsv));
+		return -EIO;
+	}
+
+	return 0;
+}
+
 /* Stop the function and discard inherited MMIO configuration. */
 static int sdxi_fn_prepare(struct sdxi_dev *sdxi)
 {
@@ -361,7 +391,20 @@ static int sdxi_fn_activate(struct sdxi_dev *sdxi)
 	if (err)
 		return err;
 
-	return 0;
+	/*
+	 * SDXI 1.0 4.1.8.9: Set MMIO_CTL0.fn_gsr to GSRV_ACTIVE and
+	 * wait for MMIO_STS0.fn_gsv to reach GSV_ACTIVE or GSV_ERROR.
+	 *
+	 * We haven't queued any descriptors to the admin context at
+	 * this point, so the admin context jump start (SDXI 1.0
+	 * 4.1.8.10.b) is accomplished later when other
+	 * code starts client contexts.
+	 */
+	err = sdxi_dev_start(sdxi);
+	if (err)
+		sdxi_dev_stop(sdxi);
+
+	return err;
 }
 
 static int sdxi_device_init(struct sdxi_dev *sdxi)
@@ -407,6 +450,14 @@ int sdxi_register(struct device *dev, const struct sdxi_bus_ops *ops)
 }
 EXPORT_SYMBOL_NS_GPL(sdxi_register, "SDXI");
 
+void sdxi_unregister(struct device *dev)
+{
+	struct sdxi_dev *sdxi = dev_get_drvdata(dev);
+
+	sdxi_dev_stop(sdxi);
+}
+EXPORT_SYMBOL_NS_GPL(sdxi_unregister, "SDXI");
+
 MODULE_AUTHOR("Wei Huang");
 MODULE_AUTHOR("Nathan Lynch");
 MODULE_DESCRIPTION("SDXI core");
diff --git a/drivers/dma/sdxi/pci.c b/drivers/dma/sdxi/pci.c
index 6a293db69199..d196ab5cf6f1 100644
--- a/drivers/dma/sdxi/pci.c
+++ b/drivers/dma/sdxi/pci.c
@@ -99,6 +99,7 @@ static int sdxi_pci_probe(struct pci_dev *pdev,
 
 static void sdxi_pci_remove(struct pci_dev *pdev)
 {
+	sdxi_unregister(&pdev->dev);
 	pci_disable_sriov(pdev);
 }
 
diff --git a/drivers/dma/sdxi/sdxi.h b/drivers/dma/sdxi/sdxi.h
index 525473029987..631a605ea62f 100644
--- a/drivers/dma/sdxi/sdxi.h
+++ b/drivers/dma/sdxi/sdxi.h
@@ -68,6 +68,7 @@ struct sdxi_dev {
 };
 
 int sdxi_register(struct device *dev, const struct sdxi_bus_ops *ops);
+void sdxi_unregister(struct device *dev);
 
 static inline u64 sdxi_read64(const struct sdxi_dev *sdxi, enum sdxi_reg reg)
 {

-- 
2.56.0

[PATCH v4 12/25] dmaengine: sdxi: Add client context allocation API

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Allocate and publish client contexts, track their IDs in a per-device
xarray, and release them during device teardown.

Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/context.c | 95 ++++++++++++++++++++++++++++++++++++++++++++++
 drivers/dma/sdxi/context.h | 12 ++++++
 drivers/dma/sdxi/device.c  |  4 ++
 drivers/dma/sdxi/sdxi.h    |  2 +
 4 files changed, 113 insertions(+)

diff --git a/drivers/dma/sdxi/context.c b/drivers/dma/sdxi/context.c
index ebce30dc462a..f52479ecf96a 100644
--- a/drivers/dma/sdxi/context.c
+++ b/drivers/dma/sdxi/context.c
@@ -44,6 +44,10 @@ static void sdxi_free_cxt(struct sdxi_cxt *cxt)
 {
 	struct sdxi_dev *sdxi = cxt->sdxi;
 
+	/* Release the id if this is a client context. */
+	if (cxt->id)
+		WARN_ON(xa_erase(&sdxi->client_cxts, cxt->id) != cxt);
+
 	if (cxt->cxt_ctl)
 		dma_pool_free(sdxi->cxt_ctl_pool, cxt->cxt_ctl,
 			      cxt->cxt_ctl_dma);
@@ -248,3 +252,94 @@ int sdxi_admin_cxt_init(struct sdxi_dev *sdxi)
 
 	return devm_add_action_or_reset(sdxi->dev, free_admin_cxt, sdxi);
 }
+
+/*
+ * Temporary owner for context id until it can be assigned to a
+ * context object; enables scope-based cleanup.
+ */
+struct sdxi_cxt_id {
+	struct sdxi_dev *sdxi;
+	u16 index;
+};
+
+static void sdxi_cxt_id_dtor(const struct sdxi_cxt_id *cxt_id)
+{
+	if (cxt_id->index == 0)
+		return;
+	WARN_ON(xa_erase(&cxt_id->sdxi->client_cxts, cxt_id->index) != NULL);
+}
+
+static struct sdxi_cxt_id sdxi_cxt_id_ctor(struct sdxi_dev *sdxi)
+{
+	struct xa_limit limit = XA_LIMIT(1, sdxi->max_cxtid);
+	u32 index;
+
+	return (struct sdxi_cxt_id) {
+		.sdxi = sdxi,
+		.index = xa_alloc(&sdxi->client_cxts, &index, NULL,
+				  limit, GFP_KERNEL) ? 0 : (u16)index,
+	};
+}
+
+DEFINE_CLASS(sdxi_cxt_id, struct sdxi_cxt_id, sdxi_cxt_id_dtor(&_T),
+	     sdxi_cxt_id_ctor(sdxi), struct sdxi_dev *sdxi)
+
+static bool sdxi_cxt_id_valid(const struct sdxi_cxt_id *cxt_id)
+{
+	return cxt_id->index > 0;
+}
+
+/*
+ * Transfer the reserved ID to the context and store its pointer in client_cxts.
+ * sdxi_free_cxt() releases the ID.
+ */
+static void sdxi_cxt_id_assign(struct sdxi_cxt *cxt, struct sdxi_cxt_id *cxt_id)
+{
+	/* We reserved the space in the constructor so this should not fail. */
+	WARN_ON(xa_store(&cxt_id->sdxi->client_cxts,
+			 cxt_id->index, cxt, GFP_KERNEL));
+	cxt->id = cxt_id->index;
+	cxt_id->index = 0;
+}
+
+/*
+ * Allocate a context for in-kernel use. Starting the context is the
+ * caller's responsibility.
+ */
+struct sdxi_cxt *sdxi_cxt_new(struct sdxi_dev *sdxi)
+{
+	/*
+	 * Ensure an ID is available before allocating memory for the
+	 * context and its control structures.
+	 */
+	CLASS(sdxi_cxt_id, id)(sdxi);
+	if (!sdxi_cxt_id_valid(&id))
+		return NULL;
+
+	struct sdxi_cxt *cxt __free(sdxi_cxt) = sdxi_alloc_cxt(sdxi);
+	if (!cxt)
+		return NULL;
+
+	sdxi_cxt_id_assign(cxt, &id);
+
+	cxt->sq->doorbell = sdxi->dbs + cxt->id * sdxi->db_stride;
+
+	if (sdxi_publish_cxt(cxt))
+		return NULL;
+
+	return_ptr(cxt);
+}
+
+/*
+ * Release all tracked client contexts during device teardown. Hardware access
+ * to client resources must have ceased before release.
+ */
+void sdxi_client_cxts_exit(struct sdxi_dev *sdxi)
+{
+	struct sdxi_cxt *cxt;
+	unsigned long index;
+
+	xa_for_each(&sdxi->client_cxts, index, cxt)
+		sdxi_free_cxt(cxt);
+	xa_destroy(&sdxi->client_cxts);
+}
diff --git a/drivers/dma/sdxi/context.h b/drivers/dma/sdxi/context.h
index ae1584d00b54..28cd7ed401f8 100644
--- a/drivers/dma/sdxi/context.h
+++ b/drivers/dma/sdxi/context.h
@@ -39,5 +39,17 @@ struct sdxi_cxt {
 };
 
 int sdxi_admin_cxt_init(struct sdxi_dev *sdxi);
+struct sdxi_cxt *sdxi_cxt_new(struct sdxi_dev *sdxi);
+void sdxi_client_cxts_exit(struct sdxi_dev *sdxi);
+
+static inline struct sdxi_cxt *to_admin_cxt(const struct sdxi_cxt *cxt)
+{
+	return cxt->sdxi->admin_cxt;
+}
+
+static inline bool sdxi_cxt_is_admin(const struct sdxi_cxt *cxt)
+{
+	return cxt == to_admin_cxt(cxt);
+}
 
 #endif /* DMA_SDXI_CONTEXT_H */
diff --git a/drivers/dma/sdxi/device.c b/drivers/dma/sdxi/device.c
index d5a5b8d43391..8786be71d3e3 100644
--- a/drivers/dma/sdxi/device.c
+++ b/drivers/dma/sdxi/device.c
@@ -19,6 +19,7 @@
 #include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/time.h>
+#include <linux/xarray.h>
 
 #include "context.h"
 #include "hw.h"
@@ -436,6 +437,7 @@ int sdxi_register(struct device *dev, const struct sdxi_bus_ops *ops)
 
 	sdxi->dev = dev;
 	sdxi->bus_ops = ops;
+	xa_init_flags(&sdxi->client_cxts, XA_FLAGS_ALLOC1);
 	dev_set_drvdata(dev, sdxi);
 
 	err = sdxi->bus_ops->init(sdxi);
@@ -454,7 +456,9 @@ void sdxi_unregister(struct device *dev)
 {
 	struct sdxi_dev *sdxi = dev_get_drvdata(dev);
 
+	/* Stop/reset is best effort if the hardware cannot be quiesced. */
 	sdxi_dev_stop(sdxi);
+	sdxi_client_cxts_exit(sdxi);
 }
 EXPORT_SYMBOL_NS_GPL(sdxi_unregister, "SDXI");
 
diff --git a/drivers/dma/sdxi/sdxi.h b/drivers/dma/sdxi/sdxi.h
index 631a605ea62f..aaa43eab00b3 100644
--- a/drivers/dma/sdxi/sdxi.h
+++ b/drivers/dma/sdxi/sdxi.h
@@ -12,6 +12,7 @@
 #include <linux/dev_printk.h>
 #include <linux/io.h>
 #include <linux/types.h>
+#include <linux/xarray.h>
 
 #include "mmio.h"
 
@@ -61,6 +62,7 @@ struct sdxi_dev {
 	struct dma_pool *cxt_ctl_pool;
 
 	struct sdxi_cxt *admin_cxt;
+	struct xarray client_cxts; /* context id -> (struct sdxi_cxt *) */
 
 	bool has_mailbox;
 

-- 
2.56.0

[PATCH v4 13/25] dmaengine: sdxi: Add descriptor ring management

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Introduce software state and reservation helpers for driving an SDXI
submission queue. A spinlock serializes space checks and write-index
updates; blocking reservations use a bounded, killable wait.

Provide exact-size and bounded-size reservations, with descriptor access
through a ring slot carrying its index and ring state. Reservations
publish the write index and cannot be rolled back, so callers must
populate and publish every reserved slot.

Allocate and initialize ring state with each administrative or client
context, and free it during context release.

The hardware read index bounds descriptor reuse. Completion handlers
wake blocked reservations as hardware consumes slots.

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/Makefile  |   3 +-
 drivers/dma/sdxi/context.c |   9 +++
 drivers/dma/sdxi/context.h |   2 +
 drivers/dma/sdxi/ring.c    | 143 +++++++++++++++++++++++++++++++++++++++++++++
 drivers/dma/sdxi/ring.h    |  94 +++++++++++++++++++++++++++++
 5 files changed, 250 insertions(+), 1 deletion(-)

diff --git a/drivers/dma/sdxi/Makefile b/drivers/dma/sdxi/Makefile
index cdf8a455077b..bfcb443f1e64 100644
--- a/drivers/dma/sdxi/Makefile
+++ b/drivers/dma/sdxi/Makefile
@@ -2,7 +2,8 @@
 obj-$(CONFIG_SDXI_CORE) += sdxi-core.o
 sdxi-core-y := \
 	context.o     \
-	device.o
+	device.o      \
+	ring.o
 
 obj-$(CONFIG_SDXI_PCI) += sdxi-pci.o
 sdxi-pci-y := pci.o
diff --git a/drivers/dma/sdxi/context.c b/drivers/dma/sdxi/context.c
index f52479ecf96a..b26eec3805fb 100644
--- a/drivers/dma/sdxi/context.c
+++ b/drivers/dma/sdxi/context.c
@@ -23,6 +23,7 @@
 
 #include "context.h"
 #include "hw.h"
+#include "ring.h"
 #include "sdxi.h"
 #include "submission.h"
 
@@ -56,6 +57,7 @@ static void sdxi_free_cxt(struct sdxi_cxt *cxt)
 				  cxt->akey_table, cxt->akey_table_dma);
 	if (cxt->sq)
 		sdxi_sq_free(sdxi, cxt->sq);
+	kfree(cxt->ring_state);
 	kfree(cxt);
 }
 
@@ -72,6 +74,10 @@ static struct sdxi_cxt *sdxi_alloc_cxt(struct sdxi_dev *sdxi)
 
 	cxt->sdxi = sdxi;
 
+	cxt->ring_state = kzalloc_obj(*cxt->ring_state, GFP_KERNEL);
+	if (!cxt->ring_state)
+		return NULL;
+
 	cxt->sq = sdxi_sq_alloc(sdxi, DEFAULT_DESC_RING_ENTRIES);
 	if (!cxt->sq)
 		return NULL;
@@ -243,6 +249,7 @@ int sdxi_admin_cxt_init(struct sdxi_dev *sdxi)
 	sq->cxt_sts.state = FIELD_PREP(SDXI_CXT_STS_STATE, CXTV_RUN);
 	cxt->id = SDXI_ADMIN_CXT_ID;
 	cxt->sq->doorbell = sdxi->dbs + cxt->id * sdxi->db_stride;
+	sdxi_ring_state_init(cxt->ring_state, sq);
 
 	err = sdxi_publish_cxt(cxt);
 	if (err)
@@ -324,6 +331,8 @@ struct sdxi_cxt *sdxi_cxt_new(struct sdxi_dev *sdxi)
 
 	cxt->sq->doorbell = sdxi->dbs + cxt->id * sdxi->db_stride;
 
+	sdxi_ring_state_init(cxt->ring_state, cxt->sq);
+
 	if (sdxi_publish_cxt(cxt))
 		return NULL;
 
diff --git a/drivers/dma/sdxi/context.h b/drivers/dma/sdxi/context.h
index 28cd7ed401f8..14ce4f5f9b74 100644
--- a/drivers/dma/sdxi/context.h
+++ b/drivers/dma/sdxi/context.h
@@ -36,6 +36,8 @@ struct sdxi_cxt {
 	dma_addr_t akey_table_dma;
 
 	struct sdxi_sq *sq;
+
+	struct sdxi_ring_state *ring_state;
 };
 
 int sdxi_admin_cxt_init(struct sdxi_dev *sdxi);
diff --git a/drivers/dma/sdxi/ring.c b/drivers/dma/sdxi/ring.c
new file mode 100644
index 000000000000..a15c8751a7be
--- /dev/null
+++ b/drivers/dma/sdxi/ring.c
@@ -0,0 +1,143 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * SDXI descriptor ring reservations and write index management
+ *
+ * Copyright Advanced Micro Devices, Inc.
+ */
+#include <kunit/test-bug.h>
+#include <kunit/visibility.h>
+#include <linux/io-64-nonatomic-lo-hi.h>
+#include <linux/jiffies.h>
+#include <linux/lockdep.h>
+#include <linux/range.h>
+#include <linux/sched.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
+#include <linux/wait.h>
+#include <asm/barrier.h>
+#include <asm/byteorder.h>
+#include <asm/div64.h>
+#include <asm/rwonce.h>
+
+#include "ring.h"
+#include "hw.h"
+#include "submission.h"
+
+/*
+ * Initialize ring management state from a submission queue. Caller is
+ * responsible for allocating and initializing @sq.
+ */
+void sdxi_ring_state_init(struct sdxi_ring_state *rs, struct sdxi_sq *sq)
+{
+	WARN_ON_ONCE(!sq);
+	/*
+	 * See SDXI 1.0 Table 3-1 Memory Structure Summary. Minimum
+	 * descriptor ring size in bytes is 64KB; thus 1024 64-byte
+	 * entries.
+	 */
+	WARN_ON_ONCE(sq->count < SZ_1K);
+
+	*rs = (typeof(*rs)) {
+		.sq = sq,
+	};
+	spin_lock_init(&rs->lock);
+	init_waitqueue_head(&rs->wqh);
+}
+EXPORT_SYMBOL_IF_KUNIT(sdxi_ring_state_init);
+
+/*
+ * Non-blocking ring reservation. Callers must handle ring full (-EBUSY).
+ *
+ * Reserving publishes the write index before the caller has written the
+ * descriptors. Callers are required to publish every reserved slot.
+ */
+int sdxi_ring_try_reserve_range(struct sdxi_ring_state *rs, size_t min_nr,
+				size_t max_nr, struct sdxi_ring_resv *resv)
+{
+	struct sdxi_sq *sq = rs->sq;
+	u64 ridx, widx, available;
+	size_t nr;
+
+	if (!min_nr || min_nr > max_nr || max_nr > sq->count) {
+		WARN_ONCE(!kunit_get_current_test(),
+			  "Reservation of size %zu..%zu requested from ring of size %u\n",
+			  min_nr, max_nr, sq->count);
+		return -EINVAL;
+	}
+
+	guard(spinlock_irqsave)(&rs->lock);
+	/* SDXI 1.0a 5.3: Read_Index advances at execution start. */
+	ridx = le64_to_cpu(READ_ONCE(sq->cxt_sts.read_index));
+	widx = le64_to_cpu(sq->write_index);
+	if (ridx > widx)
+		return -EIO;
+	if (widx - ridx > sq->count)
+		return -EIO;
+
+	available = sq->count - (widx - ridx);
+	if (available < min_nr)
+		return -EBUSY;
+	nr = min_t(u64, max_nr, available);
+
+	/* The device reads this; it must not observe a torn value. */
+	WRITE_ONCE(sq->write_index, cpu_to_le64(widx + nr));
+	*resv = (typeof(*resv)) {
+		.rs = rs,
+		.range = {
+			.start = widx,
+			.end = widx + nr - 1,
+		},
+	};
+	return 0;
+}
+EXPORT_SYMBOL_IF_KUNIT(sdxi_ring_try_reserve_range);
+
+int sdxi_ring_try_reserve(struct sdxi_ring_state *rs, size_t nr,
+			  struct sdxi_ring_resv *resv)
+{
+	return sdxi_ring_try_reserve_range(rs, nr, nr, resv);
+}
+EXPORT_SYMBOL_IF_KUNIT(sdxi_ring_try_reserve);
+
+/*
+ * Wait up to one second for ring space; return early on success, a
+ * non-transient error, or a fatal signal.
+ */
+int sdxi_ring_reserve(struct sdxi_ring_state *rs, size_t nr,
+		      struct sdxi_ring_resv *resv)
+{
+	int ret, left;
+
+	left = wait_event_killable_timeout(rs->wqh,
+					   (ret = sdxi_ring_try_reserve(rs, nr, resv)) != -EBUSY,
+					   HZ);
+
+	/* success or hard error from try_reserve() */
+	if (left > 0)
+		return ret;
+
+	/* timed out */
+	if (left == 0)
+		return -ETIMEDOUT;
+
+	/* killed: -ERESTARTSYS */
+	return left;
+}
+
+/* Completion code should call this whenever descriptors have been consumed. */
+void sdxi_ring_wake_up(struct sdxi_ring_state *rs)
+{
+	wake_up_all(&rs->wqh);
+}
+
+/*
+ * Map a logical (unwrapped) ring index onto its entry. The ring size is
+ * not constrained to a power of two, hence the division.
+ */
+struct sdxi_desc *sdxi_ring_desc(const struct sdxi_ring_state *rs, u64 index)
+{
+	struct sdxi_sq *sq = rs->sq;
+
+	return &sq->ring[do_div(index, sq->count)];
+}
+EXPORT_SYMBOL_IF_KUNIT(sdxi_ring_desc);
diff --git a/drivers/dma/sdxi/ring.h b/drivers/dma/sdxi/ring.h
new file mode 100644
index 000000000000..fb6c7ab36b57
--- /dev/null
+++ b/drivers/dma/sdxi/ring.h
@@ -0,0 +1,94 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright Advanced Micro Devices, Inc. */
+#ifndef DMA_SDXI_RING_H
+#define DMA_SDXI_RING_H
+
+#include <linux/bug.h>
+#include <linux/io-64-nonatomic-lo-hi.h>
+#include <linux/range.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
+#include <linux/wait.h>
+#include <asm/barrier.h>
+#include <asm/byteorder.h>
+#include <asm/div64.h>
+#include <asm/rwonce.h>
+
+#include "hw.h"
+
+struct sdxi_sq;
+
+/*
+ * struct sdxi_ring_state - Software state for driving a submission queue.
+ *
+ * @lock: Guards the write index.
+ * @sq: The submission queue being driven.
+ * @wqh: Pending reservations.
+ */
+struct sdxi_ring_state {
+	spinlock_t lock;
+	struct sdxi_sq *sq;
+	wait_queue_head_t wqh;
+};
+
+/*
+ * A claim on a contiguous span of the ring's logical index space. The
+ * indexes are monotonic and unwrapped; sdxi_ring_desc() maps them onto
+ * ring entries.
+ */
+struct sdxi_ring_resv {
+	const struct sdxi_ring_state *rs;
+	struct range range;
+};
+
+void sdxi_ring_state_init(struct sdxi_ring_state *ring, struct sdxi_sq *sq);
+void sdxi_ring_wake_up(struct sdxi_ring_state *rs);
+int sdxi_ring_reserve(struct sdxi_ring_state *ring, size_t nr,
+		      struct sdxi_ring_resv *resv);
+int sdxi_ring_try_reserve(struct sdxi_ring_state *ring, size_t nr,
+			  struct sdxi_ring_resv *resv);
+int sdxi_ring_try_reserve_range(struct sdxi_ring_state *rs, size_t min_nr,
+				size_t max_nr, struct sdxi_ring_resv *resv);
+struct sdxi_desc *sdxi_ring_desc(const struct sdxi_ring_state *rs, u64 index);
+
+/* Number of slots held by @resv. */
+static inline size_t sdxi_ring_resv_count(const struct sdxi_ring_resv *resv)
+{
+	return range_len(&resv->range);
+}
+
+/*
+ * Logical index of slot @i of @resv, suitable for passing to
+ * sdxi_ring_desc(). @i must be less than sdxi_ring_resv_count().
+ */
+static inline u64 sdxi_ring_resv_index(const struct sdxi_ring_resv *resv,
+				       unsigned int i)
+{
+	/* Keep a caller's mistake inside the slots it owns. */
+	if (WARN_ON_ONCE(i >= sdxi_ring_resv_count(resv)))
+		return resv->range.start;
+	return resv->range.start + i;
+}
+
+/* A reserved descriptor slot identified by its ring and logical index. */
+struct sdxi_slot {
+	const struct sdxi_ring_state *rs;
+	u64 index;
+};
+
+/* Slot @i of @resv. @i must be less than sdxi_ring_resv_count(). */
+static inline struct sdxi_slot sdxi_ring_resv_slot(const struct sdxi_ring_resv *resv,
+						   unsigned int i)
+{
+	return (struct sdxi_slot) {
+		.rs = resv->rs,
+		.index = sdxi_ring_resv_index(resv, i),
+	};
+}
+
+static inline struct sdxi_desc *sdxi_slot_desc(struct sdxi_slot slot)
+{
+	return sdxi_ring_desc(slot.rs, slot.index);
+}
+
+#endif /* DMA_SDXI_RING_H */

-- 
2.56.0

[PATCH v4 14/25] dmaengine: sdxi: Add unit tests for descriptor ring reservations

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Add hardware-independent KUnit tests for full-ring and single-slot
reservations, invalid counts, inconsistent indices, insufficient space,
and bounded reservations and descriptor access after ring wrap.

Include the SDXI KUnit configuration and test build support. Run with:

  tools/testing/kunit/kunit.py run \
    --kunitconfig=drivers/dma/sdxi/.kunitconfig

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/.kunitconfig |   4 ++
 drivers/dma/sdxi/Kconfig      |  10 +++
 drivers/dma/sdxi/Makefile     |   3 +
 drivers/dma/sdxi/ring_kunit.c | 147 ++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 164 insertions(+)

diff --git a/drivers/dma/sdxi/.kunitconfig b/drivers/dma/sdxi/.kunitconfig
new file mode 100644
index 000000000000..16a1ae04b156
--- /dev/null
+++ b/drivers/dma/sdxi/.kunitconfig
@@ -0,0 +1,4 @@
+CONFIG_KUNIT=y
+CONFIG_DMADEVICES=y
+CONFIG_SDXI_CORE=y
+CONFIG_SDXI_KUNIT_TEST=y
diff --git a/drivers/dma/sdxi/Kconfig b/drivers/dma/sdxi/Kconfig
index b91b44231a04..7217375c9216 100644
--- a/drivers/dma/sdxi/Kconfig
+++ b/drivers/dma/sdxi/Kconfig
@@ -26,3 +26,13 @@ config SDXI_PCI
 
 	  To compile this driver as a module, choose M here: the module
 	  will be named "sdxi-pci".
+
+config SDXI_KUNIT_TEST
+	tristate "SDXI unit tests" if !KUNIT_ALL_TESTS
+	depends on SDXI_CORE && KUNIT
+	default KUNIT_ALL_TESTS
+	help
+	  KUnit tests for parts of the SDXI driver. Does not require
+	  SDXI hardware.
+
+	  If unsure, say N.
diff --git a/drivers/dma/sdxi/Makefile b/drivers/dma/sdxi/Makefile
index bfcb443f1e64..00e3f1cb0808 100644
--- a/drivers/dma/sdxi/Makefile
+++ b/drivers/dma/sdxi/Makefile
@@ -7,3 +7,6 @@ sdxi-core-y := \
 
 obj-$(CONFIG_SDXI_PCI) += sdxi-pci.o
 sdxi-pci-y := pci.o
+
+obj-$(CONFIG_SDXI_KUNIT_TEST) += \
+	ring_kunit.o
diff --git a/drivers/dma/sdxi/ring_kunit.c b/drivers/dma/sdxi/ring_kunit.c
new file mode 100644
index 000000000000..7df85fc5549c
--- /dev/null
+++ b/drivers/dma/sdxi/ring_kunit.c
@@ -0,0 +1,147 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * SDXI descriptor ring management tests.
+ *
+ * Copyright Advanced Micro Devices, Inc.
+ */
+#include <kunit/device.h>
+#include <kunit/test-bug.h>
+#include <kunit/test.h>
+#include <linux/container_of.h>
+#include <linux/dma-mapping.h>
+#include <linux/module.h>
+#include <linux/overflow.h>
+#include <linux/packing.h>
+#include <linux/string.h>
+
+#include "ring.h"
+#include "submission.h"
+
+MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
+
+/* A submission queue backed by plain memory; no device or DMA required. */
+static struct sdxi_sq *alloc_sq(struct kunit *t, u32 count)
+{
+	struct sdxi_sq *sq;
+
+	sq = kunit_kmalloc(t, struct_size(sq, ring, count),
+			   GFP_KERNEL | __GFP_ZERO);
+	if (sq)
+		sq->count = count;
+	return sq;
+}
+
+static void valid(struct kunit *t)
+{
+	struct sdxi_ring_state r;
+	struct sdxi_ring_resv resv;
+	struct sdxi_sq *sq;
+
+	sq = alloc_sq(t, SZ_1K);
+	KUNIT_ASSERT_NOT_NULL(t, sq);
+
+	sdxi_ring_state_init(&r, sq);
+
+	KUNIT_EXPECT_EQ(t, sdxi_ring_try_reserve(&r, sq->count, &resv), 0);
+	KUNIT_EXPECT_EQ(t, resv.range.start, 0);
+	KUNIT_EXPECT_EQ(t, resv.range.end, sq->count - 1);
+	KUNIT_EXPECT_EQ(t, sdxi_ring_resv_count(&resv), sq->count);
+	KUNIT_EXPECT_EQ(t, le64_to_cpu(sq->write_index), sq->count);
+
+	/* A ring-sized reservation maps slot i onto ring entry i. */
+	for (unsigned int i = 0; i < sq->count; i++) {
+		KUNIT_EXPECT_EQ(t, sdxi_ring_resv_index(&resv, i), i);
+		KUNIT_EXPECT_PTR_EQ(t, sdxi_ring_desc(&r, i), &sq->ring[i]);
+	}
+
+	sq->cxt_sts.read_index = cpu_to_le64(1);
+	KUNIT_EXPECT_EQ(t, sdxi_ring_try_reserve(&r, 1, &resv), 0);
+	KUNIT_EXPECT_EQ(t, le64_to_cpu(sq->write_index), sq->count + 1);
+	KUNIT_EXPECT_EQ(t, sdxi_ring_resv_count(&resv), 1);
+
+	/* The index space is unwrapped: index sq->count is entry 0. */
+	KUNIT_EXPECT_EQ(t, sdxi_ring_resv_index(&resv, 0), sq->count);
+	KUNIT_EXPECT_PTR_EQ(t, sdxi_ring_desc(&r, sq->count), &sq->ring[0]);
+}
+
+static void invalid(struct kunit *t)
+{
+	struct sdxi_ring_state rs;
+	struct sdxi_ring_resv resv;
+	struct sdxi_sq *sq;
+
+	sq = alloc_sq(t, SZ_1K);
+	KUNIT_ASSERT_NOT_NULL(t, sq);
+
+	sdxi_ring_state_init(&rs, sq);
+
+	KUNIT_EXPECT_EQ(t, sdxi_ring_try_reserve(&rs, 0, &resv), -EINVAL);
+	KUNIT_EXPECT_EQ(t, sdxi_ring_try_reserve(&rs, sq->count + 1, &resv), -EINVAL);
+
+	sq->cxt_sts.read_index = cpu_to_le64(1);
+	KUNIT_EXPECT_EQ(t, sdxi_ring_try_reserve(&rs, 1, &resv), -EIO);
+
+	sq->cxt_sts.read_index = 0;
+	sq->write_index = cpu_to_le64(sq->count);
+	sdxi_ring_state_init(&rs, sq);
+	KUNIT_EXPECT_EQ(t, sdxi_ring_try_reserve(&rs, 1, &resv), -EBUSY);
+
+	sq->cxt_sts.read_index = cpu_to_le64(sq->count);
+	sq->write_index = cpu_to_le64(sq->count + 1);
+	sdxi_ring_state_init(&rs, sq);
+	KUNIT_EXPECT_EQ(t, sdxi_ring_try_reserve(&rs, sq->count, &resv), -EBUSY);
+}
+
+static void bounded_reservations(struct kunit *t)
+{
+	struct sdxi_ring_state rs;
+	struct sdxi_ring_resv resv;
+	struct sdxi_sq *sq = alloc_sq(t, SZ_1K);
+
+	KUNIT_ASSERT_NOT_NULL(t, sq);
+	sdxi_ring_state_init(&rs, sq);
+	KUNIT_EXPECT_EQ(t, sdxi_ring_try_reserve_range(&rs, 0, 2, &resv), -EINVAL);
+	KUNIT_EXPECT_EQ(t, sdxi_ring_try_reserve_range(&rs, 3, 2, &resv), -EINVAL);
+	KUNIT_EXPECT_EQ(t, sdxi_ring_try_reserve_range(&rs, 2, sq->count + 1, &resv),
+			-EINVAL);
+
+	sq->write_index = cpu_to_le64(sq->count - 2);
+	KUNIT_ASSERT_EQ(t, sdxi_ring_try_reserve_range(&rs, 2, 10, &resv), 0);
+	KUNIT_EXPECT_EQ(t, sdxi_ring_resv_count(&resv), 2);
+	KUNIT_EXPECT_EQ(t, resv.range.start, sq->count - 2);
+	KUNIT_EXPECT_EQ(t, sdxi_ring_try_reserve_range(&rs, 2, 10, &resv), -EBUSY);
+	sq->cxt_sts.read_index = cpu_to_le64(1);
+	KUNIT_EXPECT_EQ(t, sdxi_ring_try_reserve_range(&rs, 2, 10, &resv), -EBUSY);
+	sq->cxt_sts.read_index = cpu_to_le64(5);
+	KUNIT_ASSERT_EQ(t, sdxi_ring_try_reserve_range(&rs, 2, 10, &resv), 0);
+	KUNIT_EXPECT_EQ(t, sdxi_ring_resv_count(&resv), 5);
+	KUNIT_EXPECT_PTR_EQ(t, sdxi_slot_desc(sdxi_ring_resv_slot(&resv, 0)),
+			    &sq->ring[0]);
+}
+
+static struct kunit_case testcases[] = {
+	KUNIT_CASE(valid),
+	KUNIT_CASE(invalid),
+	KUNIT_CASE(bounded_reservations),
+	{}
+};
+
+static int setup_device(struct kunit *t)
+{
+	struct device *dev = kunit_device_register(t, "sdxi-mock-device");
+
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(t, dev);
+	t->priv = dev;
+	return 0;
+}
+
+static struct kunit_suite generic_desc_ts = {
+	.name = "SDXI descriptor ring management",
+	.test_cases = testcases,
+	.init = setup_device,
+};
+kunit_test_suite(generic_desc_ts);
+
+MODULE_DESCRIPTION("SDXI descriptor ring tests");
+MODULE_AUTHOR("Nathan Lynch");
+MODULE_LICENSE("GPL");

-- 
2.56.0

[PATCH v4 15/25] dmaengine: sdxi: Per-context access key (AKey) table entry allocator

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Each SDXI context has a table of access keys (AKeys). SDXI descriptors
submitted to a context may refer to an AKey associated with that
context by its index in the table. AKeys describe properties of the
access that the descriptor is to perform, such as PASID or a target
SDXI function, or an interrupt to trigger.

Use a per-context IDA to keep track of used entries in the table.
Provide sdxi_alloc_akey(), which claims an AKey table entry for the
caller to program directly; sdxi_akey_index(), which returns the
entry's index for programming into descriptors the caller intends to
submit; and sdxi_free_akey(), which clears the entry and makes it
available again.

The DMAengine provider uses one AKey for copy-buffer access and completion
interrupts.

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/context.c |  3 +++
 drivers/dma/sdxi/context.h | 24 ++++++++++++++++++++++++
 2 files changed, 27 insertions(+)

diff --git a/drivers/dma/sdxi/context.c b/drivers/dma/sdxi/context.c
index b26eec3805fb..956bb06f9499 100644
--- a/drivers/dma/sdxi/context.c
+++ b/drivers/dma/sdxi/context.c
@@ -15,6 +15,7 @@
 #include <linux/dma-mapping.h>
 #include <linux/dmapool.h>
 #include <linux/errno.h>
+#include <linux/idr.h>
 #include <linux/iommu.h>
 #include <linux/slab.h>
 #include <linux/types.h>
@@ -57,6 +58,7 @@ static void sdxi_free_cxt(struct sdxi_cxt *cxt)
 				  cxt->akey_table, cxt->akey_table_dma);
 	if (cxt->sq)
 		sdxi_sq_free(sdxi, cxt->sq);
+	ida_destroy(&cxt->akey_ida);
 	kfree(cxt->ring_state);
 	kfree(cxt);
 }
@@ -73,6 +75,7 @@ static struct sdxi_cxt *sdxi_alloc_cxt(struct sdxi_dev *sdxi)
 		return NULL;
 
 	cxt->sdxi = sdxi;
+	ida_init(&cxt->akey_ida);
 
 	cxt->ring_state = kzalloc_obj(*cxt->ring_state, GFP_KERNEL);
 	if (!cxt->ring_state)
diff --git a/drivers/dma/sdxi/context.h b/drivers/dma/sdxi/context.h
index 14ce4f5f9b74..cad5a153ec22 100644
--- a/drivers/dma/sdxi/context.h
+++ b/drivers/dma/sdxi/context.h
@@ -6,7 +6,10 @@
 #ifndef DMA_SDXI_CONTEXT_H
 #define DMA_SDXI_CONTEXT_H
 
+#include <linux/array_size.h>
 #include <linux/dma-mapping.h>
+#include <linux/idr.h>
+#include <linux/string.h>
 #include <linux/types.h>
 
 #include "hw.h"
@@ -32,6 +35,7 @@ struct sdxi_cxt {
 	struct sdxi_cxt_ctl *cxt_ctl;
 	dma_addr_t cxt_ctl_dma;
 
+	struct ida akey_ida;
 	struct sdxi_akey_table *akey_table;
 	dma_addr_t akey_table_dma;
 
@@ -54,4 +58,24 @@ static inline bool sdxi_cxt_is_admin(const struct sdxi_cxt *cxt)
 	return cxt == to_admin_cxt(cxt);
 }
 
+static inline struct sdxi_akey_ent *sdxi_alloc_akey(struct sdxi_cxt *cxt)
+{
+	unsigned int max = ARRAY_SIZE(cxt->akey_table->entry) - 1;
+	int idx = ida_alloc_max(&cxt->akey_ida, max, GFP_KERNEL);
+
+	return idx < 0 ? NULL : &cxt->akey_table->entry[idx];
+}
+
+static inline unsigned int sdxi_akey_index(const struct sdxi_cxt *cxt,
+					   const struct sdxi_akey_ent *akey)
+{
+	return akey - &cxt->akey_table->entry[0];
+}
+
+static inline void sdxi_free_akey(struct sdxi_cxt *cxt, struct sdxi_akey_ent *akey)
+{
+	memset(akey, 0, sizeof(*akey));
+	ida_free(&cxt->akey_ida, sdxi_akey_index(cxt, akey));
+}
+
 #endif /* DMA_SDXI_CONTEXT_H */

-- 
2.56.0

[PATCH v4 16/25] dmaengine: sdxi: Add completion status block API

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Introduce completion status handling and a fixed-capacity pool owned by
each context. Only the 32-byte-aligned status blocks are DMA-visible;
a private bitmap and spinlock track claims independently of descriptor
ring indices.

Preallocate storage during context allocation. Claiming and initializing
a block cannot allocate memory and fails only when capacity is exhausted.
Provide a bounded, killable wait for administrative callers, and wake
waiters when blocks are returned. Release blocks only after consumption,
unpublished rollback, or hardware quiescence; destroy the pool only when
hardware and software no longer access it.

Represent a completion-status block by its CPU pointer and DMA address.
Check the little-endian signal with READ_ONCE(), then use dma_rmb() before
consuming error status or operation results. Polling returns success,
-EIO for a terminal error, or -ETIMEDOUT without relinquishing ownership.

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/Makefile     |   1 +
 drivers/dma/sdxi/completion.c | 152 ++++++++++++++++++++++++++++++++++++++++++
 drivers/dma/sdxi/completion.h |  48 +++++++++++++
 drivers/dma/sdxi/context.c    |   7 ++
 drivers/dma/sdxi/context.h    |   4 ++
 drivers/dma/sdxi/hw.h         |  15 +++++
 6 files changed, 227 insertions(+)

diff --git a/drivers/dma/sdxi/Makefile b/drivers/dma/sdxi/Makefile
index 00e3f1cb0808..eacad504a816 100644
--- a/drivers/dma/sdxi/Makefile
+++ b/drivers/dma/sdxi/Makefile
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_SDXI_CORE) += sdxi-core.o
 sdxi-core-y := \
+	completion.o  \
 	context.o     \
 	device.o      \
 	ring.o
diff --git a/drivers/dma/sdxi/completion.c b/drivers/dma/sdxi/completion.c
new file mode 100644
index 000000000000..c07a0fc101c8
--- /dev/null
+++ b/drivers/dma/sdxi/completion.c
@@ -0,0 +1,152 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * SDXI Descriptor Completion Status Block handling.
+ *
+ * Copyright Advanced Micro Devices, Inc.
+ */
+#include <linux/bitfield.h>
+#include <linux/bitmap.h>
+#include <linux/dma-mapping.h>
+#include <linux/iopoll.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/wait.h>
+
+#include "completion.h"
+#include "hw.h"
+
+/* Only the status blocks themselves are visible to hardware. */
+struct sdxi_cst_pool {
+	struct sdxi_cst_blk *blocks;
+	dma_addr_t dma;
+	unsigned int count;
+	spinlock_t lock;
+	wait_queue_head_t wait;
+	unsigned long *allocated;
+};
+
+static void sdxi_cst_pool_init(struct sdxi_cst_pool *pool, unsigned int count)
+{
+	pool->count = count;
+	spin_lock_init(&pool->lock);
+	init_waitqueue_head(&pool->wait);
+}
+
+struct sdxi_cst_pool *sdxi_cst_pool_create(struct sdxi_dev *sdxi,
+					unsigned int count)
+{
+	struct sdxi_cst_pool *pool;
+
+	if (!count)
+		return NULL;
+
+	pool = kzalloc_obj(*pool, GFP_KERNEL);
+	if (!pool)
+		return NULL;
+
+	pool->allocated = bitmap_zalloc(count, GFP_KERNEL);
+	if (!pool->allocated) {
+		kfree(pool);
+		return NULL;
+	}
+
+	/* SDXI 1.0a 6.1.1 requires a 32-byte-aligned completion block. */
+	pool->blocks = dma_alloc_coherent(sdxi->dev,
+					 array_size(count, sizeof(*pool->blocks)),
+					 &pool->dma, GFP_KERNEL);
+	if (!pool->blocks) {
+		bitmap_free(pool->allocated);
+		kfree(pool);
+		return NULL;
+	}
+
+	sdxi_cst_pool_init(pool, count);
+	return pool;
+}
+
+void sdxi_cst_pool_destroy(struct sdxi_dev *sdxi, struct sdxi_cst_pool *pool)
+{
+	dma_free_coherent(sdxi->dev, array_size(pool->count, sizeof(*pool->blocks)),
+			  pool->blocks, pool->dma);
+	bitmap_free(pool->allocated);
+	kfree(pool);
+}
+
+bool sdxi_cst_pool_get(struct sdxi_cst_pool *pool, struct sdxi_completion *comp)
+{
+	unsigned long flags;
+	unsigned int index;
+
+	spin_lock_irqsave(&pool->lock, flags);
+	index = find_first_zero_bit(pool->allocated, pool->count);
+	if (index < pool->count)
+		__set_bit(index, pool->allocated);
+	spin_unlock_irqrestore(&pool->lock, flags);
+	if (index == pool->count)
+		return false;
+
+	comp->cst_blk = &pool->blocks[index];
+	comp->cst_blk_dma = pool->dma + index * sizeof(*pool->blocks);
+	sdxi_cst_blk_init(comp->cst_blk);
+	return true;
+}
+
+int sdxi_cst_pool_get_wait(struct sdxi_cst_pool *pool,
+			 struct sdxi_completion *comp)
+{
+	long ret;
+
+	ret = wait_event_killable_timeout(pool->wait,
+					 sdxi_cst_pool_get(pool, comp), HZ);
+	if (ret < 0)
+		return ret;
+	return ret ? 0 : -ETIMEDOUT;
+}
+
+void sdxi_cst_pool_put(struct sdxi_cst_pool *pool, struct sdxi_cst_blk *cst)
+{
+	unsigned int index = cst - pool->blocks;
+	unsigned long flags;
+
+	if (WARN_ON_ONCE(index >= pool->count))
+		return;
+
+	spin_lock_irqsave(&pool->lock, flags);
+	WARN_ON_ONCE(!__test_and_clear_bit(index, pool->allocated));
+	spin_unlock_irqrestore(&pool->lock, flags);
+	wake_up_all(&pool->wait);
+}
+
+
+enum sdxi_completion_state sdxi_cst_blk_check(const struct sdxi_cst_blk *cst)
+{
+	u64 signal = le64_to_cpu(READ_ONCE(cst->signal));
+
+	if (signal != SDXI_CST_BLK_SIGNAL_TERMINAL)
+		return SDXI_COMPLETION_PENDING;
+	/*
+	 * The final ER state (and the operation's effects) are
+	 * guaranteed visible only after a cleared signal is observed.
+	 * See SDXI 1.0 5.6 Memory Consistency Model.
+	 */
+	dma_rmb();
+
+	if (FIELD_GET(SDXI_CST_BLK_ER, le32_to_cpu(cst->flags)))
+		return SDXI_COMPLETION_ERROR;
+
+	return SDXI_COMPLETION_DONE;
+}
+
+int sdxi_cst_blk_poll(const struct sdxi_cst_blk *cst)
+{
+	enum sdxi_completion_state state;
+	int ret;
+
+	ret = read_poll_timeout(sdxi_cst_blk_check, state,
+				state != SDXI_COMPLETION_PENDING,
+				10, USEC_PER_SEC, false, cst);
+	if (ret)
+		return ret;		/* -ETIMEDOUT */
+
+	return state == SDXI_COMPLETION_ERROR ? -EIO : 0;
+}
diff --git a/drivers/dma/sdxi/completion.h b/drivers/dma/sdxi/completion.h
new file mode 100644
index 000000000000..f91f44b9b516
--- /dev/null
+++ b/drivers/dma/sdxi/completion.h
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright Advanced Micro Devices, Inc. */
+#ifndef DMA_SDXI_COMPLETION_H
+#define DMA_SDXI_COMPLETION_H
+
+#include <linux/compiler_attributes.h>
+#include "sdxi.h"
+#include "hw.h"
+
+
+/* A completion status block and the device address the hardware needs. */
+struct sdxi_completion {
+	struct sdxi_cst_blk *cst_blk;
+	dma_addr_t cst_blk_dma;
+};
+
+struct sdxi_cst_pool;
+
+/* Software completion states derived from CST_BLK. */
+enum sdxi_completion_state {
+	SDXI_COMPLETION_PENDING,
+	SDXI_COMPLETION_DONE,
+	SDXI_COMPLETION_ERROR,
+};
+
+struct sdxi_cst_pool *sdxi_cst_pool_create(struct sdxi_dev *sdxi,
+					unsigned int count);
+/* Hardware and software must no longer access any blocks before destruction. */
+void sdxi_cst_pool_destroy(struct sdxi_dev *sdxi, struct sdxi_cst_pool *pool);
+/* Claims and initializes a block; false means the fixed pool is exhausted. */
+bool sdxi_cst_pool_get(struct sdxi_cst_pool *pool, struct sdxi_completion *comp);
+int sdxi_cst_pool_get_wait(struct sdxi_cst_pool *pool,
+			 struct sdxi_completion *comp);
+/* Release only after consumption, unpublished rollback, or hardware quiescence. */
+void sdxi_cst_pool_put(struct sdxi_cst_pool *pool, struct sdxi_cst_blk *cst);
+
+
+static inline void sdxi_cst_blk_init(struct sdxi_cst_blk *cst)
+{
+	*cst = (typeof(*cst)) {
+		.signal = cpu_to_le64(SDXI_CST_BLK_SIGNAL_INIT),
+	};
+}
+
+int __must_check sdxi_cst_blk_poll(const struct sdxi_cst_blk *cst);
+enum sdxi_completion_state sdxi_cst_blk_check(const struct sdxi_cst_blk *cst);
+
+#endif /* DMA_SDXI_COMPLETION_H */
diff --git a/drivers/dma/sdxi/context.c b/drivers/dma/sdxi/context.c
index 956bb06f9499..60cc812df204 100644
--- a/drivers/dma/sdxi/context.c
+++ b/drivers/dma/sdxi/context.c
@@ -22,6 +22,7 @@
 #include <asm/barrier.h>
 #include <asm/rwonce.h>
 
+#include "completion.h"
 #include "context.h"
 #include "hw.h"
 #include "ring.h"
@@ -56,6 +57,8 @@ static void sdxi_free_cxt(struct sdxi_cxt *cxt)
 	if (cxt->akey_table)
 		dma_free_coherent(sdxi->dev, sizeof(*cxt->akey_table),
 				  cxt->akey_table, cxt->akey_table_dma);
+	if (cxt->cst_pool)
+		sdxi_cst_pool_destroy(sdxi, cxt->cst_pool);
 	if (cxt->sq)
 		sdxi_sq_free(sdxi, cxt->sq);
 	ida_destroy(&cxt->akey_ida);
@@ -85,6 +88,10 @@ static struct sdxi_cxt *sdxi_alloc_cxt(struct sdxi_dev *sdxi)
 	if (!cxt->sq)
 		return NULL;
 
+	cxt->cst_pool = sdxi_cst_pool_create(sdxi, DEFAULT_DESC_RING_ENTRIES);
+	if (!cxt->cst_pool)
+		return NULL;
+
 	cxt->akey_table = dma_alloc_coherent(dev, sizeof(*cxt->akey_table),
 					     &cxt->akey_table_dma, GFP_KERNEL);
 	if (!cxt->akey_table)
diff --git a/drivers/dma/sdxi/context.h b/drivers/dma/sdxi/context.h
index cad5a153ec22..94e0fc45f281 100644
--- a/drivers/dma/sdxi/context.h
+++ b/drivers/dma/sdxi/context.h
@@ -16,6 +16,9 @@
 #include "sdxi.h"
 #include "submission.h"
 
+struct sdxi_completion;
+struct sdxi_cst_pool;
+
 /* Use a 4 KiB AKey table, the minimum architectural size. */
 struct sdxi_akey_table {
 	struct sdxi_akey_ent entry[SZ_4K / sizeof(struct sdxi_akey_ent)];
@@ -40,6 +43,7 @@ struct sdxi_cxt {
 	dma_addr_t akey_table_dma;
 
 	struct sdxi_sq *sq;
+	struct sdxi_cst_pool *cst_pool;
 
 	struct sdxi_ring_state *ring_state;
 };
diff --git a/drivers/dma/sdxi/hw.h b/drivers/dma/sdxi/hw.h
index b333e091c583..c1bff631a5ae 100644
--- a/drivers/dma/sdxi/hw.h
+++ b/drivers/dma/sdxi/hw.h
@@ -110,6 +110,21 @@ struct sdxi_akey_ent {
 } __packed __aligned(16);
 static_assert(sizeof(struct sdxi_akey_ent) == 16);
 
+/* SDXI 1.0 Table 6-4: CST_BLK (Completion Status Block) */
+struct sdxi_cst_blk {
+	__le64 signal;
+	__le32 flags;
+#define SDXI_CST_BLK_ER BIT(31)
+	__u8 rsvd_0[20];
+} __packed __aligned(32);
+static_assert(sizeof(struct sdxi_cst_blk) == 32);
+
+/* SDXI 1.0 4.4.2 Completion-Status Modes */
+enum sdxi_cst_blk_signal {
+	SDXI_CST_BLK_SIGNAL_TERMINAL = 0,
+	SDXI_CST_BLK_SIGNAL_INIT     = 1,
+};
+
 struct sdxi_desc {
 	union {
 		/*

-- 
2.56.0

[PATCH v4 17/25] dmaengine: sdxi: Add unit tests for completion pool ownership

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Add hardware-independent KUnit coverage for completion-pool exhaustion
across a bitmap word boundary, arbitrary release and reuse, reinitialization
of reused status blocks. Verify that a claimed block remains unavailable
until explicitly released, including after a simulated late device completion.
Also cover release of an unpublished claim.

Provide test allocation helpers and KUnit-only exports so the tests exercise
the production claim and release paths without a coherent-DMA device.

Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/Makefile           |   1 +
 drivers/dma/sdxi/completion.c       |  37 ++++++++++
 drivers/dma/sdxi/completion.h       |   4 ++
 drivers/dma/sdxi/completion_kunit.c | 131 ++++++++++++++++++++++++++++++++++++
 4 files changed, 173 insertions(+)

diff --git a/drivers/dma/sdxi/Makefile b/drivers/dma/sdxi/Makefile
index eacad504a816..d12eebe03870 100644
--- a/drivers/dma/sdxi/Makefile
+++ b/drivers/dma/sdxi/Makefile
@@ -10,4 +10,5 @@ obj-$(CONFIG_SDXI_PCI) += sdxi-pci.o
 sdxi-pci-y := pci.o
 
 obj-$(CONFIG_SDXI_KUNIT_TEST) += \
+	completion_kunit.o    \
 	ring_kunit.o
diff --git a/drivers/dma/sdxi/completion.c b/drivers/dma/sdxi/completion.c
index c07a0fc101c8..dacd1cac6303 100644
--- a/drivers/dma/sdxi/completion.c
+++ b/drivers/dma/sdxi/completion.c
@@ -4,6 +4,7 @@
  *
  * Copyright Advanced Micro Devices, Inc.
  */
+#include <kunit/visibility.h>
 #include <linux/bitfield.h>
 #include <linux/bitmap.h>
 #include <linux/dma-mapping.h>
@@ -90,6 +91,7 @@ bool sdxi_cst_pool_get(struct sdxi_cst_pool *pool, struct sdxi_completion *comp)
 	sdxi_cst_blk_init(comp->cst_blk);
 	return true;
 }
+EXPORT_SYMBOL_IF_KUNIT(sdxi_cst_pool_get);
 
 int sdxi_cst_pool_get_wait(struct sdxi_cst_pool *pool,
 			 struct sdxi_completion *comp)
@@ -116,7 +118,42 @@ void sdxi_cst_pool_put(struct sdxi_cst_pool *pool, struct sdxi_cst_blk *cst)
 	spin_unlock_irqrestore(&pool->lock, flags);
 	wake_up_all(&pool->wait);
 }
+EXPORT_SYMBOL_IF_KUNIT(sdxi_cst_pool_put);
 
+#if IS_ENABLED(CONFIG_SDXI_KUNIT_TEST)
+/* Exercise allocation without requiring a device capable of coherent DMA. */
+struct sdxi_cst_pool *sdxi_cst_pool_test_create(unsigned int count)
+{
+	struct sdxi_cst_pool *pool;
+
+	pool = kzalloc_obj(*pool, GFP_KERNEL);
+	if (!pool)
+		return NULL;
+	pool->allocated = bitmap_zalloc(count, GFP_KERNEL);
+	if (!pool->allocated) {
+		kfree(pool);
+		return NULL;
+	}
+	pool->blocks = kcalloc(count, sizeof(*pool->blocks), GFP_KERNEL);
+	if (!pool->blocks) {
+		bitmap_free(pool->allocated);
+		kfree(pool);
+		return NULL;
+	}
+	pool->dma = 0x1000;
+	sdxi_cst_pool_init(pool, count);
+	return pool;
+}
+EXPORT_SYMBOL_IF_KUNIT(sdxi_cst_pool_test_create);
+
+void sdxi_cst_pool_test_destroy(struct sdxi_cst_pool *pool)
+{
+	kfree(pool->blocks);
+	bitmap_free(pool->allocated);
+	kfree(pool);
+}
+EXPORT_SYMBOL_IF_KUNIT(sdxi_cst_pool_test_destroy);
+#endif
 
 enum sdxi_completion_state sdxi_cst_blk_check(const struct sdxi_cst_blk *cst)
 {
diff --git a/drivers/dma/sdxi/completion.h b/drivers/dma/sdxi/completion.h
index f91f44b9b516..18c89701d1d6 100644
--- a/drivers/dma/sdxi/completion.h
+++ b/drivers/dma/sdxi/completion.h
@@ -34,6 +34,10 @@ int sdxi_cst_pool_get_wait(struct sdxi_cst_pool *pool,
 /* Release only after consumption, unpublished rollback, or hardware quiescence. */
 void sdxi_cst_pool_put(struct sdxi_cst_pool *pool, struct sdxi_cst_blk *cst);
 
+#if IS_ENABLED(CONFIG_SDXI_KUNIT_TEST)
+struct sdxi_cst_pool *sdxi_cst_pool_test_create(unsigned int count);
+void sdxi_cst_pool_test_destroy(struct sdxi_cst_pool *pool);
+#endif
 
 static inline void sdxi_cst_blk_init(struct sdxi_cst_blk *cst)
 {
diff --git a/drivers/dma/sdxi/completion_kunit.c b/drivers/dma/sdxi/completion_kunit.c
new file mode 100644
index 000000000000..326e3d84da0d
--- /dev/null
+++ b/drivers/dma/sdxi/completion_kunit.c
@@ -0,0 +1,131 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright Advanced Micro Devices, Inc. */
+#include <kunit/test.h>
+#include <linux/module.h>
+
+#include "completion.h"
+
+MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
+
+static void destroy_pool(void *pool)
+{
+	sdxi_cst_pool_test_destroy(pool);
+}
+
+static struct sdxi_cst_pool *alloc_pool(struct kunit *test, unsigned int count)
+{
+	struct sdxi_cst_pool *pool = sdxi_cst_pool_test_create(count);
+
+	if (!pool || kunit_add_action_or_reset(test, destroy_pool, pool))
+		return NULL;
+	return pool;
+}
+
+static void exhaustion(struct kunit *test)
+{
+	/* Cross a bitmap word boundary without filling the final word. */
+	const unsigned int count = BITS_PER_LONG + 3;
+	struct sdxi_completion *held, extra = {};
+	struct sdxi_cst_pool *pool;
+
+	pool = alloc_pool(test, count);
+	KUNIT_ASSERT_NOT_NULL(test, pool);
+	held = kunit_kcalloc(test, count, sizeof(*held), GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, held);
+	for (unsigned int i = 0; i < count; i++) {
+		KUNIT_ASSERT_TRUE(test, sdxi_cst_pool_get(pool, &held[i]));
+		KUNIT_EXPECT_EQ(test, held[i].cst_blk_dma % 32, 0);
+		KUNIT_EXPECT_EQ(test, le64_to_cpu(held[i].cst_blk->signal),
+				(u64)SDXI_CST_BLK_SIGNAL_INIT);
+		for (unsigned int j = 0; j < i; j++) {
+			KUNIT_EXPECT_PTR_NE(test, held[i].cst_blk, held[j].cst_blk);
+			KUNIT_EXPECT_NE(test, held[i].cst_blk_dma, held[j].cst_blk_dma);
+		}
+	}
+	KUNIT_EXPECT_FALSE(test, sdxi_cst_pool_get(pool, &extra));
+	KUNIT_EXPECT_PTR_EQ(test, extra.cst_blk, NULL);
+	for (unsigned int i = 0; i < count; i++)
+		sdxi_cst_pool_put(pool, held[i].cst_blk);
+}
+
+static void arbitrary_release(struct kunit *test)
+{
+	struct sdxi_completion held[5], comp;
+	struct sdxi_cst_pool *pool;
+	const unsigned int order[] = { 3, 1, 4, 0, 2 };
+
+	pool = alloc_pool(test, ARRAY_SIZE(held));
+	KUNIT_ASSERT_NOT_NULL(test, pool);
+	for (unsigned int i = 0; i < ARRAY_SIZE(held); i++)
+		KUNIT_ASSERT_TRUE(test, sdxi_cst_pool_get(pool, &held[i]));
+
+	/* A released block is reusable while all other owners retain theirs. */
+	for (unsigned int i = 0; i < ARRAY_SIZE(order); i++) {
+		unsigned int index = order[i];
+
+		sdxi_cst_pool_put(pool, held[index].cst_blk);
+		KUNIT_ASSERT_TRUE(test, sdxi_cst_pool_get(pool, &comp));
+		KUNIT_EXPECT_PTR_EQ(test, comp.cst_blk, held[index].cst_blk);
+		KUNIT_EXPECT_EQ(test, comp.cst_blk_dma, held[index].cst_blk_dma);
+		KUNIT_EXPECT_FALSE(test, sdxi_cst_pool_get(pool, &comp));
+	}
+	for (unsigned int i = 0; i < ARRAY_SIZE(held); i++)
+		sdxi_cst_pool_put(pool, held[i].cst_blk);
+}
+
+static void reuse_initializes_status(struct kunit *test)
+{
+	const struct sdxi_cst_blk initial = {
+		.signal = cpu_to_le64(SDXI_CST_BLK_SIGNAL_INIT),
+	};
+	struct sdxi_completion first, second;
+	struct sdxi_cst_pool *pool;
+
+	pool = alloc_pool(test, 1);
+	KUNIT_ASSERT_NOT_NULL(test, pool);
+	KUNIT_ASSERT_TRUE(test, sdxi_cst_pool_get(pool, &first));
+	memset(first.cst_blk, 0xff, sizeof(*first.cst_blk));
+	first.cst_blk->signal = cpu_to_le64(SDXI_CST_BLK_SIGNAL_TERMINAL);
+	sdxi_cst_pool_put(pool, first.cst_blk);
+	KUNIT_ASSERT_TRUE(test, sdxi_cst_pool_get(pool, &second));
+	KUNIT_EXPECT_PTR_EQ(test, first.cst_blk, second.cst_blk);
+	KUNIT_EXPECT_MEMEQ(test, second.cst_blk, &initial, sizeof(initial));
+	sdxi_cst_pool_put(pool, second.cst_blk);
+}
+
+static void retained_completion(struct kunit *test)
+{
+	struct sdxi_completion timed_out, other;
+	struct sdxi_cst_pool *pool;
+
+	pool = alloc_pool(test, 1);
+	KUNIT_ASSERT_NOT_NULL(test, pool);
+	KUNIT_ASSERT_TRUE(test, sdxi_cst_pool_get(pool, &timed_out));
+	KUNIT_EXPECT_FALSE(test, sdxi_cst_pool_get(pool, &other));
+	/* Even a late completion remains allocated until its owner releases it. */
+	timed_out.cst_blk->signal = cpu_to_le64(SDXI_CST_BLK_SIGNAL_TERMINAL);
+	KUNIT_EXPECT_FALSE(test, sdxi_cst_pool_get(pool, &other));
+	sdxi_cst_pool_put(pool, timed_out.cst_blk);
+	KUNIT_ASSERT_TRUE(test, sdxi_cst_pool_get(pool, &other));
+	/* A never-published claim can be returned without a terminal signal. */
+	sdxi_cst_pool_put(pool, other.cst_blk);
+	KUNIT_ASSERT_TRUE(test, sdxi_cst_pool_get(pool, &other));
+	sdxi_cst_pool_put(pool, other.cst_blk);
+}
+
+static struct kunit_case completion_testcases[] = {
+	KUNIT_CASE(exhaustion),
+	KUNIT_CASE(arbitrary_release),
+	KUNIT_CASE(reuse_initializes_status),
+	KUNIT_CASE(retained_completion),
+	{}
+};
+
+static struct kunit_suite completion_tests = {
+	.name = "SDXI completion pool",
+	.test_cases = completion_testcases,
+};
+kunit_test_suite(completion_tests);
+
+MODULE_DESCRIPTION("SDXI completion pool tests");
+MODULE_LICENSE("GPL");

-- 
2.56.0

[PATCH v4 18/25] dmaengine: sdxi: Add context submission helper

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Add sdxi_cxt_submit() to consolidate optional fence and completion-status
block setup, valid-bit publication, and doorbell ringing. An optional
struct sdxi_completion input identifies storage already claimed and
initialized by the caller.

Introduce generic descriptor helpers for attaching completion storage,
setting completion-status mode, fencing, and publishing the valid bit.
Publishing orders prior descriptor writes before transferring ownership
to hardware; the helpers warn if a caller attempts to modify an already
valid descriptor.

Provide sdxi_cxt_attach_cst() for attaching completion storage without
publishing the descriptor, and sdxi_cxt_kick() for ringing a context
doorbell.

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/context.c    | 36 ++++++++++++++++++++++++
 drivers/dma/sdxi/context.h    | 28 +++++++++++++++++++
 drivers/dma/sdxi/descriptor.h | 64 +++++++++++++++++++++++++++++++++++++++++++
 drivers/dma/sdxi/hw.h         | 10 +++++++
 4 files changed, 138 insertions(+)

diff --git a/drivers/dma/sdxi/context.c b/drivers/dma/sdxi/context.c
index 60cc812df204..679c5cfab19d 100644
--- a/drivers/dma/sdxi/context.c
+++ b/drivers/dma/sdxi/context.c
@@ -24,6 +24,7 @@
 
 #include "completion.h"
 #include "context.h"
+#include "descriptor.h"
 #include "hw.h"
 #include "ring.h"
 #include "sdxi.h"
@@ -362,3 +363,38 @@ void sdxi_client_cxts_exit(struct sdxi_dev *sdxi)
 		sdxi_free_cxt(cxt);
 	xa_destroy(&sdxi->client_cxts);
 }
+
+/*
+ * Attach a claimed, initialized completion block to a descriptor slot.
+ * The descriptor is not published; the caller sets the valid bit (and
+ * kicks) once the operation is fully encoded.
+ */
+void sdxi_cxt_attach_cst(struct sdxi_slot slot,
+			 const struct sdxi_completion *comp)
+{
+	struct sdxi_desc *desc = sdxi_slot_desc(slot);
+
+	/*
+	 * We use simple (non-atomic) completion status mode only; set the
+	 * descriptor's CSR bit. See SDXI 1.0 4.4.2 Completion-Status Modes.
+	 */
+	sdxi_desc_set_csr(desc);
+	sdxi_desc_set_csb(desc, comp->cst_blk_dma);
+}
+
+void sdxi_cxt_submit(struct sdxi_cxt *cxt, struct sdxi_slot slot,
+		     const struct sdxi_completion *comp, enum sdxi_submit_flags flags)
+{
+	struct sdxi_desc *desc = sdxi_slot_desc(slot);
+
+	if (flags & SDXI_CXT_SUBMIT_FENCE)
+		sdxi_desc_set_fence(desc);
+
+	if (comp)
+		sdxi_cxt_attach_cst(slot, comp);
+
+	sdxi_desc_make_valid(desc);
+
+	if (flags & SDXI_CXT_SUBMIT_KICK)
+		sdxi_cxt_kick(cxt);
+}
diff --git a/drivers/dma/sdxi/context.h b/drivers/dma/sdxi/context.h
index 94e0fc45f281..f048e298f6de 100644
--- a/drivers/dma/sdxi/context.h
+++ b/drivers/dma/sdxi/context.h
@@ -9,10 +9,12 @@
 #include <linux/array_size.h>
 #include <linux/dma-mapping.h>
 #include <linux/idr.h>
+#include <linux/io.h>
 #include <linux/string.h>
 #include <linux/types.h>
 
 #include "hw.h"
+#include "ring.h"
 #include "sdxi.h"
 #include "submission.h"
 
@@ -52,6 +54,16 @@ int sdxi_admin_cxt_init(struct sdxi_dev *sdxi);
 struct sdxi_cxt *sdxi_cxt_new(struct sdxi_dev *sdxi);
 void sdxi_client_cxts_exit(struct sdxi_dev *sdxi);
 
+enum sdxi_submit_flags {
+	SDXI_CXT_SUBMIT_KICK  = BIT(0),
+	SDXI_CXT_SUBMIT_FENCE = BIT(1),
+};
+
+void sdxi_cxt_attach_cst(struct sdxi_slot slot,
+			 const struct sdxi_completion *comp);
+void sdxi_cxt_submit(struct sdxi_cxt *cxt, struct sdxi_slot slot,
+		     const struct sdxi_completion *comp, enum sdxi_submit_flags flags);
+
 static inline struct sdxi_cxt *to_admin_cxt(const struct sdxi_cxt *cxt)
 {
 	return cxt->sdxi->admin_cxt;
@@ -62,6 +74,22 @@ static inline bool sdxi_cxt_is_admin(const struct sdxi_cxt *cxt)
 	return cxt == to_admin_cxt(cxt);
 }
 
+static inline void sdxi_cxt_kick(struct sdxi_cxt *cxt)
+{
+	/*
+	 * Ring the context's doorbell with its current write index.
+	 *
+	 * It's OK to locklessly sample the index for this purpose: it
+	 * monotonically increases and the implementation is required
+	 * to ignore out-of-order doorbell values (SDXI 1.0 5.2.1
+	 * Multi-Producer Enqueue).
+	 *
+	 * writeq() orders the descriptors' valid bits ahead of the
+	 * doorbell, so no explicit barrier is needed here.
+	 */
+	writeq(le64_to_cpu(READ_ONCE(cxt->sq->write_index)), cxt->sq->doorbell);
+}
+
 static inline struct sdxi_akey_ent *sdxi_alloc_akey(struct sdxi_cxt *cxt)
 {
 	unsigned int max = ARRAY_SIZE(cxt->akey_table->entry) - 1;
diff --git a/drivers/dma/sdxi/descriptor.h b/drivers/dma/sdxi/descriptor.h
new file mode 100644
index 000000000000..b19d5d329e43
--- /dev/null
+++ b/drivers/dma/sdxi/descriptor.h
@@ -0,0 +1,64 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef DMA_SDXI_DESCRIPTOR_H
+#define DMA_SDXI_DESCRIPTOR_H
+
+/*
+ * Facilities for encoding SDXI descriptors.
+ *
+ * Copyright Advanced Micro Devices, Inc.
+ */
+
+#include <linux/bitfield.h>
+#include <linux/ratelimit.h>
+#include <linux/types.h>
+#include <asm/byteorder.h>
+
+#include "hw.h"
+
+static inline void sdxi_desc_vl_expect(const struct sdxi_desc *desc, bool expected)
+{
+	u8 vl = FIELD_GET(SDXI_DSC_VL, le32_to_cpu(desc->opcode));
+
+	WARN_RATELIMIT(vl != expected, "expected vl=%u but got %u\n", expected, vl);
+}
+
+static inline void sdxi_desc_set_csb(struct sdxi_desc *desc, dma_addr_t addr)
+{
+	sdxi_desc_vl_expect(desc, 0);
+	desc->csb_ptr = cpu_to_le64(FIELD_PREP(SDXI_DSC_CSB_PTR, addr >> 5));
+}
+
+static inline void sdxi_desc_make_valid(struct sdxi_desc *desc)
+{
+	u32 opcode = le32_to_cpu(desc->opcode);
+
+	sdxi_desc_vl_expect(desc, 0);
+	FIELD_MODIFY(SDXI_DSC_VL, &opcode, 1);
+	/*
+	 * Once vl is set, no more modifications to the descriptor
+	 * payload are allowed. Ensure the vl update is ordered after
+	 * all other initialization of the descriptor.
+	 */
+	dma_wmb();
+	WRITE_ONCE(desc->opcode, cpu_to_le32(opcode));
+}
+
+static inline void sdxi_desc_set_fence(struct sdxi_desc *desc)
+{
+	u32 opcode = le32_to_cpu(desc->opcode);
+
+	sdxi_desc_vl_expect(desc, 0);
+	FIELD_MODIFY(SDXI_DSC_FE, &opcode, 1);
+	desc->opcode = cpu_to_le32(opcode);
+}
+
+static inline void sdxi_desc_set_csr(struct sdxi_desc *desc)
+{
+	u32 opcode = le32_to_cpu(desc->opcode);
+
+	sdxi_desc_vl_expect(desc, 0);
+	FIELD_MODIFY(SDXI_DSC_CSR, &opcode, 1);
+	desc->opcode = cpu_to_le32(opcode);
+}
+
+#endif /* DMA_SDXI_DESCRIPTOR_H */
diff --git a/drivers/dma/sdxi/hw.h b/drivers/dma/sdxi/hw.h
index c1bff631a5ae..edc43190148c 100644
--- a/drivers/dma/sdxi/hw.h
+++ b/drivers/dma/sdxi/hw.h
@@ -136,6 +136,16 @@ struct sdxi_desc {
 			__u8 operation[52];
 			__le64 csb_ptr;
 		);
+
+/* For opcode field */
+#define SDXI_DSC_VL  BIT(0)
+#define SDXI_DSC_SE  BIT(1)
+#define SDXI_DSC_FE  BIT(2)
+#define SDXI_DSC_CSR BIT(4)
+
+/* For csb_ptr field */
+#define SDXI_DSC_CSB_PTR GENMASK_ULL(63, 5)
+
 	};
 } __packed __aligned(64);
 static_assert(sizeof(struct sdxi_desc) == 64);

-- 
2.56.0

[PATCH v4 19/25] dmaengine: sdxi: Encode context start, stop, update, and sync descriptors

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Add layouts and encoders for administrative context start, stop, update,
and synchronization descriptors. Parameters select the context range,
update level, and synchronization filter. Encoders set mandatory fields
and produce little-endian descriptors with VL clear; callers attach
completion storage and publish them.

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/Makefile     |   1 +
 drivers/dma/sdxi/descriptor.c | 119 ++++++++++++++++++++++++++++++++++++++++++
 drivers/dma/sdxi/descriptor.h |  58 ++++++++++++++++++++
 drivers/dma/sdxi/hw.h         |  77 +++++++++++++++++++++++++++
 4 files changed, 255 insertions(+)

diff --git a/drivers/dma/sdxi/Makefile b/drivers/dma/sdxi/Makefile
index d12eebe03870..e9c180ea6fa4 100644
--- a/drivers/dma/sdxi/Makefile
+++ b/drivers/dma/sdxi/Makefile
@@ -3,6 +3,7 @@ obj-$(CONFIG_SDXI_CORE) += sdxi-core.o
 sdxi-core-y := \
 	completion.o  \
 	context.o     \
+	descriptor.o  \
 	device.o      \
 	ring.o
 
diff --git a/drivers/dma/sdxi/descriptor.c b/drivers/dma/sdxi/descriptor.c
new file mode 100644
index 000000000000..424453ff7ba0
--- /dev/null
+++ b/drivers/dma/sdxi/descriptor.c
@@ -0,0 +1,119 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * SDXI descriptor encoding.
+ *
+ * Copyright Advanced Micro Devices, Inc.
+ */
+
+#include <kunit/visibility.h>
+#include <linux/bitfield.h>
+#include <linux/types.h>
+#include <asm/byteorder.h>
+
+#include "hw.h"
+#include "descriptor.h"
+
+int sdxi_encode_cxt_start(struct sdxi_desc *desc,
+			  const struct sdxi_cxt_start *params)
+{
+	u64 csb_ptr;
+	u32 opcode;
+
+	opcode = (FIELD_PREP(SDXI_DSC_FE, 1) |
+		  FIELD_PREP(SDXI_DSC_SUBTYPE, SDXI_DSC_OP_SUBTYPE_CXT_START_NM) |
+		  FIELD_PREP(SDXI_DSC_TYPE, SDXI_DSC_OP_TYPE_ADMIN));
+
+	csb_ptr = FIELD_PREP(SDXI_DSC_NP, 1);
+
+	*desc = (typeof(*desc)) {
+		.cxt_start = (typeof(desc->cxt_start)) {
+			.opcode = cpu_to_le32(opcode),
+			.cxt_start = cpu_to_le16(params->range.cxt_start),
+			.cxt_end = cpu_to_le16(params->range.cxt_end),
+			.csb_ptr = cpu_to_le64(csb_ptr),
+		},
+	};
+
+	return 0;
+}
+EXPORT_SYMBOL_IF_KUNIT(sdxi_encode_cxt_start);
+
+int sdxi_encode_cxt_stop(struct sdxi_desc *desc,
+			  const struct sdxi_cxt_stop *params)
+{
+	u64 csb_ptr;
+	u32 opcode;
+
+	opcode = (FIELD_PREP(SDXI_DSC_FE, 1) |
+		  FIELD_PREP(SDXI_DSC_SUBTYPE, SDXI_DSC_OP_SUBTYPE_CXT_STOP) |
+		  FIELD_PREP(SDXI_DSC_TYPE, SDXI_DSC_OP_TYPE_ADMIN));
+
+	csb_ptr = FIELD_PREP(SDXI_DSC_NP, 1);
+
+	*desc = (typeof(*desc)) {
+		.cxt_stop = (typeof(desc->cxt_stop)) {
+			.opcode = cpu_to_le32(opcode),
+			.cxt_start = cpu_to_le16(params->range.cxt_start),
+			.cxt_end = cpu_to_le16(params->range.cxt_end),
+			.csb_ptr = cpu_to_le64(csb_ptr),
+		},
+	};
+
+	return 0;
+}
+EXPORT_SYMBOL_IF_KUNIT(sdxi_encode_cxt_stop);
+
+int sdxi_encode_cxt_upd(struct sdxi_desc *desc,
+			const struct sdxi_cxt_upd *params)
+{
+	u64 csb_ptr;
+	u32 opcode;
+	u8 cflags;
+
+	opcode = (FIELD_PREP(SDXI_DSC_SUBTYPE, SDXI_DSC_OP_SUBTYPE_CXT_UPD) |
+		  FIELD_PREP(SDXI_DSC_TYPE, SDXI_DSC_OP_TYPE_ADMIN));
+
+	cflags = FIELD_PREP(SDXI_DSC_CXT_UPD_PCD, params->level);
+
+	csb_ptr = FIELD_PREP(SDXI_DSC_NP, 1);
+
+	*desc = (typeof(*desc)) {
+		.cxt_upd = (typeof(desc->cxt_upd)) {
+			.opcode = cpu_to_le32(opcode),
+			.cflags = cflags,
+			.cxt_start = cpu_to_le16(params->range.cxt_start),
+			.cxt_end = cpu_to_le16(params->range.cxt_end),
+			.csb_ptr = cpu_to_le64(csb_ptr),
+		},
+	};
+
+	return 0;
+}
+EXPORT_SYMBOL_IF_KUNIT(sdxi_encode_cxt_upd);
+
+int sdxi_encode_sync(struct sdxi_desc *desc, const struct sdxi_sync *params)
+{
+	u64 csb_ptr;
+	u32 opcode;
+	u8 cflags;
+
+	opcode = (FIELD_PREP(SDXI_DSC_SUBTYPE, SDXI_DSC_OP_SUBTYPE_SYNC) |
+		  FIELD_PREP(SDXI_DSC_TYPE, SDXI_DSC_OP_TYPE_ADMIN));
+
+	cflags = FIELD_PREP(SDXI_DSC_SYNC_FLT, params->filter);
+
+	csb_ptr = FIELD_PREP(SDXI_DSC_NP, 1);
+
+	*desc = (typeof(*desc)) {
+		.sync = (typeof(desc->sync)) {
+			.opcode = cpu_to_le32(opcode),
+			.cflags = cflags,
+			.cxt_start = cpu_to_le16(params->range.cxt_start),
+			.cxt_end = cpu_to_le16(params->range.cxt_end),
+			.csb_ptr = cpu_to_le64(csb_ptr),
+		},
+	};
+
+	return 0;
+}
+EXPORT_SYMBOL_IF_KUNIT(sdxi_encode_sync);
diff --git a/drivers/dma/sdxi/descriptor.h b/drivers/dma/sdxi/descriptor.h
index b19d5d329e43..9138f7c85f3a 100644
--- a/drivers/dma/sdxi/descriptor.h
+++ b/drivers/dma/sdxi/descriptor.h
@@ -9,6 +9,7 @@
  */
 
 #include <linux/bitfield.h>
+#include <linux/minmax.h>
 #include <linux/ratelimit.h>
 #include <linux/types.h>
 #include <asm/byteorder.h>
@@ -61,4 +62,61 @@ static inline void sdxi_desc_set_csr(struct sdxi_desc *desc)
 	desc->opcode = cpu_to_le32(opcode);
 }
 
+struct sdxi_cxt_range {
+	u16 cxt_start;
+	u16 cxt_end;
+};
+
+static inline struct sdxi_cxt_range sdxi_cxt_range(u16 a, u16 b)
+{
+	return (struct sdxi_cxt_range) {
+		.cxt_start = min(a, b),
+		.cxt_end   = max(a, b),
+	};
+}
+
+static inline struct sdxi_cxt_range sdxi_cxt_range_single(u16 nr)
+{
+	return sdxi_cxt_range(nr, nr);
+}
+
+struct sdxi_cxt_start {
+	struct sdxi_cxt_range range;
+};
+
+int sdxi_encode_cxt_start(struct sdxi_desc *desc,
+			  const struct sdxi_cxt_start *params);
+
+struct sdxi_cxt_stop {
+	struct sdxi_cxt_range range;
+};
+
+int sdxi_encode_cxt_stop(struct sdxi_desc *desc,
+			  const struct sdxi_cxt_stop *params);
+
+struct sdxi_cxt_upd {
+	enum sdxi_cxt_upd_level {
+		SDXI_CXT_UPD_LVL_CTL = 0x4,
+		SDXI_CXT_UPD_LVL_L1  = 0x6,
+		SDXI_CXT_UPD_LVL_L2  = 0x7,
+	} level;
+	struct sdxi_cxt_range range;
+};
+
+int sdxi_encode_cxt_upd(struct sdxi_desc *desc,
+			const struct sdxi_cxt_upd *params);
+
+struct sdxi_sync {
+	enum sdxi_sync_filter  {
+		SDXI_SYNC_FLT_CXT  = 0x0,
+		SDXI_SYNC_FLT_STOP = 0x1,
+		SDXI_SYNC_FLT_AKEY = 0x2,
+		SDXI_SYNC_FLT_RKEY = 0x3,
+		SDXI_SYNC_FLT_FN   = 0x4,
+	} filter;
+	struct sdxi_cxt_range range;
+};
+
+int sdxi_encode_sync(struct sdxi_desc *desc, const struct sdxi_sync *params);
+
 #endif /* DMA_SDXI_DESCRIPTOR_H */
diff --git a/drivers/dma/sdxi/hw.h b/drivers/dma/sdxi/hw.h
index edc43190148c..51b2bcbeca90 100644
--- a/drivers/dma/sdxi/hw.h
+++ b/drivers/dma/sdxi/hw.h
@@ -142,12 +142,89 @@ struct sdxi_desc {
 #define SDXI_DSC_SE  BIT(1)
 #define SDXI_DSC_FE  BIT(2)
 #define SDXI_DSC_CSR BIT(4)
+#define SDXI_DSC_SUBTYPE GENMASK(15, 8)
+#define SDXI_DSC_TYPE    GENMASK(26, 16)
 
 /* For csb_ptr field */
+#define SDXI_DSC_NP BIT_ULL(0)
 #define SDXI_DSC_CSB_PTR GENMASK_ULL(63, 5)
 
+#define define_sdxi_dsc(tag_, name_, op_body_)				\
+	struct tag_ {							\
+		__le32 opcode;						\
+		op_body_						\
+		__le64 csb_ptr;						\
+	} __packed __aligned(64) name_;					\
+	static_assert(sizeof(struct tag_) ==				\
+		      sizeof(struct sdxi_dsc_generic));			\
+	static_assert(offsetof(struct tag_, csb_ptr) ==			\
+		      offsetof(struct sdxi_dsc_generic, csb_ptr))
+
+		/* SDXI 1.0 Table 6-14: DSC_CXT_START Descriptor Format */
+		define_sdxi_dsc(sdxi_dsc_cxt_start, cxt_start,
+			__u8 rsvd_0;
+			__u8 vflags;
+			__le16 vf_num;
+			__le16 cxt_start;
+			__le16 cxt_end;
+			__u8 rsvd_1[4];
+			__le64 db_value;
+			__u8 rsvd_2[32];
+		);
+
+		/* SDXI 1.0 Table 6-15: DSC_CXT_STOP Descriptor Format */
+		define_sdxi_dsc(sdxi_dsc_cxt_stop, cxt_stop,
+			__u8 rsvd_0;
+			__u8 vflags;
+			__le16 vf_num;
+			__le16 cxt_start;
+			__le16 cxt_end;
+			__u8 rsvd_1[44];
+		);
+
+		/* SDXI 1.0 Table 6-18: DSC_CXT_UPD Descriptor Format */
+		define_sdxi_dsc(sdxi_dsc_cxt_upd, cxt_upd,
+			__u8 cflags;
+			__u8 vflags;
+			__le16 vf_num;
+			__le16 cxt_start;
+			__le16 cxt_end;
+			__u8 rsvd_0[44];
+		);
+/* For use with cxt_upd.cflags */
+#define SDXI_DSC_CXT_UPD_PCD GENMASK(2, 0)
+
+		/* SDXI 1.0 Table 6-22: DSC_SYNC Descriptor Format */
+		define_sdxi_dsc(sdxi_dsc_sync, sync,
+			__u8 cflags;
+			__u8 vflags;
+			__le16 vf_num;
+			__le16 cxt_start;
+			__le16 cxt_end;
+			__le16 key_start;
+			__le16 key_end;
+			__u8 rsvd_0[40];
+		);
+/* For use with sync.cflags */
+#define SDXI_DSC_SYNC_FLT GENMASK(2, 0)
+
+#undef define_sdxi_dsc
 	};
 } __packed __aligned(64);
 static_assert(sizeof(struct sdxi_desc) == 64);
 
+/* SDXI 1.0 Table 6-1: SDXI Operation Groups */
+enum sdxi_dsc_type {
+	SDXI_DSC_OP_TYPE_ADMIN   = 0x002,
+};
+
+/* SDXI 1.0 Table 6-2: SDXI Operation Groups, Types, and Subtypes */
+enum sdxi_dsc_subtype {
+	/* Administrative */
+	SDXI_DSC_OP_SUBTYPE_CXT_UPD      = 0x01,
+	SDXI_DSC_OP_SUBTYPE_CXT_START_NM = 0x03,
+	SDXI_DSC_OP_SUBTYPE_CXT_STOP     = 0x04,
+	SDXI_DSC_OP_SUBTYPE_SYNC         = 0x06,
+};
+
 #endif /* DMA_SDXI_HW_H */

-- 
2.56.0

[PATCH v4 20/25] dmaengine: sdxi: Add client context release with update and sync

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Invalidate an unused, stopped client context and synchronize hardware
caches before freeing its resources. Add administrative update, reservation, and
completion-polling helpers.

Claim completion-status storage before making an irrevocable ring
reservation. Return the claim if reservation fails, and release it after
consuming a terminal result. A timeout leaves hardware ownership
unresolved, so retain that block in the administrative pool until context
teardown, even if the device writes its terminal signal later. Repeated
timeouts can exhaust the pool, causing further claims to time out rather
than reuse outstanding status blocks.

If update or synchronization fails, retain the context and its ID for
cleanup after function shutdown. Device-wide teardown frees clients after
the best-effort function stop/reset path without submitting additional
administrative operations.

Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/context.c | 128 +++++++++++++++++++++++++++++++++++++++++++++
 drivers/dma/sdxi/context.h |   6 +++
 2 files changed, 134 insertions(+)

diff --git a/drivers/dma/sdxi/context.c b/drivers/dma/sdxi/context.c
index 679c5cfab19d..5ca99ac8080c 100644
--- a/drivers/dma/sdxi/context.c
+++ b/drivers/dma/sdxi/context.c
@@ -106,6 +106,78 @@ static struct sdxi_cxt *sdxi_alloc_cxt(struct sdxi_dev *sdxi)
 	return_ptr(cxt);
 }
 
+/* Claim completion storage before making a ring reservation irrevocable. */
+static int sdxi_cxt_reserve(struct sdxi_cxt *cxt, size_t nr,
+			    struct sdxi_ring_resv *resv,
+			    struct sdxi_completion *comp)
+{
+	int err;
+
+	err = sdxi_cst_pool_get_wait(cxt->cst_pool, comp);
+	if (err)
+		return err;
+
+	err = sdxi_ring_reserve(cxt->ring_state, nr, resv);
+	if (err)
+		sdxi_cst_pool_put(cxt->cst_pool, comp->cst_blk);
+
+	return err;
+}
+
+static int sdxi_cxt_poll_completion(struct sdxi_cxt *cxt,
+				    const struct sdxi_completion *comp)
+{
+	int err = sdxi_cst_blk_poll(comp->cst_blk);
+
+	/*
+	 * A timeout does not end hardware ownership. Keep these administrative
+	 * blocks claimed until context teardown, even if they complete later.
+	 * Repeated timeouts can exhaust the pool; allocation then times out too.
+	 */
+	if (err != -ETIMEDOUT)
+		sdxi_cst_pool_put(cxt->cst_pool, comp->cst_blk);
+	sdxi_ring_wake_up(cxt->ring_state);
+
+	return err;
+}
+
+/*
+ * Signal the function that a client context's control structures have
+ * been modified so it discards any privately cached copies of them.
+ * DSC_SYNC.CXT is required before the operation can be considered
+ * complete.
+ */
+static int sdxi_cxt_update(const struct sdxi_cxt *cxt)
+{
+	struct sdxi_cxt *adm = to_admin_cxt(cxt);
+	struct sdxi_slot upd, sync;
+	struct sdxi_ring_resv resv;
+	struct sdxi_completion comp;
+	int err;
+
+	might_sleep();
+
+	err = sdxi_cxt_reserve(adm, 2, &resv, &comp);
+	if (err)
+		return err;
+
+	upd = sdxi_ring_resv_slot(&resv, 0);
+	sync = sdxi_ring_resv_slot(&resv, 1);
+
+	sdxi_encode_cxt_upd(sdxi_slot_desc(upd), &(const struct sdxi_cxt_upd) {
+			.level = SDXI_CXT_UPD_LVL_L1,
+			.range = sdxi_cxt_range_single(cxt->id),
+		});
+	sdxi_encode_sync(sdxi_slot_desc(sync), &(const struct sdxi_sync) {
+			.filter = SDXI_SYNC_FLT_CXT,
+			.range = sdxi_cxt_range_single(cxt->id),
+		});
+	sdxi_cxt_submit(adm, upd, NULL, 0);
+	sdxi_cxt_submit(adm, sync, &comp, SDXI_CXT_SUBMIT_KICK);
+
+	return sdxi_cxt_poll_completion(adm, &comp);
+}
+
 struct sdxi_cxt_ctl_cfg {
 	dma_addr_t ds_ring_ptr;
 	dma_addr_t cxt_sts_ptr;
@@ -144,6 +216,16 @@ static int configure_cxt_ctl(struct sdxi_cxt_ctl *ctl, const struct sdxi_cxt_ctl
 	return 0;
 }
 
+static void invalidate_cxt_ctl(struct sdxi_cxt_ctl *ctl)
+{
+	u64 ds_ring_ptr = le64_to_cpu(ctl->ds_ring_ptr);
+
+	FIELD_MODIFY(SDXI_CXT_CTL_VL, &ds_ring_ptr, 0);
+	WRITE_ONCE(ctl->ds_ring_ptr, cpu_to_le64(ds_ring_ptr));
+	dma_wmb();
+	*ctl = (typeof(*ctl)) { 0 };
+}
+
 /*
  * Logical representation of CXT_L1_ENT subfields.
  */
@@ -194,6 +276,17 @@ static int configure_L1_entry(struct sdxi_cxt_L1_ent *ent,
 	return 0;
 }
 
+static void invalidate_L1_entry(struct sdxi_cxt_L1_ent *ent)
+{
+	u64 cxt_ctl_ptr = le64_to_cpu(ent->cxt_ctl_ptr);
+
+	FIELD_MODIFY(SDXI_CXT_L1_ENT_VL, &cxt_ctl_ptr, 0);
+	WRITE_ONCE(ent->cxt_ctl_ptr, cpu_to_le64(cxt_ctl_ptr));
+	dma_wmb();
+	*ent = (typeof(*ent)) { 0 };
+}
+
+
 /* Publish the context control block and L1 entry to hardware. */
 static int sdxi_publish_cxt(const struct sdxi_cxt *cxt)
 {
@@ -398,3 +491,38 @@ void sdxi_cxt_submit(struct sdxi_cxt *cxt, struct sdxi_slot slot,
 	if (flags & SDXI_CXT_SUBMIT_KICK)
 		sdxi_cxt_kick(cxt);
 }
+
+/*
+ * Release an unused, stopped client context. On update/sync failure the
+ * context remains owned by the device and its resources are retained for
+ * cleanup after function shutdown; the caller must not reuse it.
+ */
+int sdxi_cxt_exit(struct sdxi_cxt *cxt)
+{
+	struct sdxi_dev *sdxi = cxt->sdxi;
+	struct sdxi_cxt_L1_ent *ent;
+	u8 state;
+	int err;
+
+	if (WARN_ON(sdxi_cxt_is_admin(cxt)))
+		return -EINVAL;
+
+	state = FIELD_GET(SDXI_CXT_STS_STATE, READ_ONCE(cxt->sq->cxt_sts.state));
+	if (state != CXTV_STOP_SW && state != CXTV_STOP_FN && state != CXTV_ERR_FN)
+		return -EBUSY;
+
+	ent = &sdxi->L1_table->entry[ID_TO_L1_INDEX(cxt->id)];
+	invalidate_L1_entry(ent);
+	invalidate_cxt_ctl(cxt->cxt_ctl);
+
+	err = sdxi_cxt_update(cxt);
+	if (err) {
+		dev_err(sdxi->dev,
+			"context %u release failed (%d), retaining resources until function shutdown\n",
+			cxt->id, err);
+		return err;
+	}
+
+	sdxi_free_cxt(cxt);
+	return 0;
+}
diff --git a/drivers/dma/sdxi/context.h b/drivers/dma/sdxi/context.h
index f048e298f6de..71eef2c6243b 100644
--- a/drivers/dma/sdxi/context.h
+++ b/drivers/dma/sdxi/context.h
@@ -52,6 +52,12 @@ struct sdxi_cxt {
 
 int sdxi_admin_cxt_init(struct sdxi_dev *sdxi);
 struct sdxi_cxt *sdxi_cxt_new(struct sdxi_dev *sdxi);
+/*
+ * Caller must exclude users and stop the client before release. An update or
+ * synchronization failure retains the invalidated context for device teardown;
+ * the caller must not reuse it.
+ */
+int sdxi_cxt_exit(struct sdxi_cxt *cxt);
 void sdxi_client_cxts_exit(struct sdxi_dev *sdxi);
 
 enum sdxi_submit_flags {

-- 
2.56.0

[PATCH v4 21/25] dmaengine: sdxi: Provide context start and stop APIs

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Implement client context start and stop by submitting administrative
descriptors. Starting waits for the start descriptor to finish; stopping
submits stop followed by synchronization and waits for the latter.

Use the administrative reservation and polling helpers to claim status
blocks before reserving descriptors, release consumed results, and retain
timed-out blocks until context teardown. Expose these operations through
struct sdxi_cxt for clients such as the DMAengine provider.

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/context.c | 61 ++++++++++++++++++++++++++++++++++++++++++++++
 drivers/dma/sdxi/context.h |  3 +++
 2 files changed, 64 insertions(+)

diff --git a/drivers/dma/sdxi/context.c b/drivers/dma/sdxi/context.c
index 5ca99ac8080c..0979fa35346b 100644
--- a/drivers/dma/sdxi/context.c
+++ b/drivers/dma/sdxi/context.c
@@ -364,6 +364,67 @@ int sdxi_admin_cxt_init(struct sdxi_dev *sdxi)
 	return devm_add_action_or_reset(sdxi->dev, free_admin_cxt, sdxi);
 }
 
+int sdxi_start_cxt(struct sdxi_cxt *cxt)
+{
+	struct sdxi_cxt *adm = to_admin_cxt(cxt);
+	struct sdxi_ring_resv resv;
+	struct sdxi_completion comp;
+	struct sdxi_slot start;
+	int err;
+
+	might_sleep();
+
+	/* Administrative context activation is controlled by the function state. */
+	if (WARN_ON(adm == cxt))
+		return -EINVAL;
+
+	err = sdxi_cxt_reserve(adm, 1, &resv, &comp);
+	if (err)
+		return err;
+
+	start = sdxi_ring_resv_slot(&resv, 0);
+	sdxi_encode_cxt_start(sdxi_slot_desc(start), &(const struct sdxi_cxt_start) {
+			.range = sdxi_cxt_range_single(cxt->id),
+		});
+	sdxi_cxt_submit(adm, start, &comp, SDXI_CXT_SUBMIT_KICK);
+
+	return sdxi_cxt_poll_completion(adm, &comp);
+}
+
+void sdxi_stop_cxt(struct sdxi_cxt *cxt)
+{
+	struct sdxi_cxt *adm = to_admin_cxt(cxt);
+	struct sdxi_slot stop, sync;
+	struct sdxi_ring_resv resv;
+	struct sdxi_completion comp;
+	int err;
+
+	might_sleep();
+
+	/* Administrative context stopping is controlled by the function state. */
+	if (WARN_ON(adm == cxt))
+		return;
+
+	err = sdxi_cxt_reserve(adm, 2, &resv, &comp);
+	if (WARN_ON_ONCE(err))
+		return;
+
+	stop = sdxi_ring_resv_slot(&resv, 0);
+	sync = sdxi_ring_resv_slot(&resv, 1);
+
+	sdxi_encode_cxt_stop(sdxi_slot_desc(stop), &(const struct sdxi_cxt_stop) {
+			.range = sdxi_cxt_range_single(cxt->id),
+		});
+	sdxi_encode_sync(sdxi_slot_desc(sync), &(const struct sdxi_sync) {
+			.filter = SDXI_SYNC_FLT_STOP,
+			.range = sdxi_cxt_range_single(cxt->id),
+		});
+	sdxi_cxt_submit(adm, stop, NULL, 0);
+	sdxi_cxt_submit(adm, sync, &comp, SDXI_CXT_SUBMIT_KICK);
+
+	WARN_ON(sdxi_cxt_poll_completion(adm, &comp));
+}
+
 /*
  * Temporary owner for context id until it can be assigned to a
  * context object; enables scope-based cleanup.
diff --git a/drivers/dma/sdxi/context.h b/drivers/dma/sdxi/context.h
index 71eef2c6243b..acf41709c4de 100644
--- a/drivers/dma/sdxi/context.h
+++ b/drivers/dma/sdxi/context.h
@@ -60,6 +60,9 @@ struct sdxi_cxt *sdxi_cxt_new(struct sdxi_dev *sdxi);
 int sdxi_cxt_exit(struct sdxi_cxt *cxt);
 void sdxi_client_cxts_exit(struct sdxi_dev *sdxi);
 
+int sdxi_start_cxt(struct sdxi_cxt *cxt);
+void sdxi_stop_cxt(struct sdxi_cxt *cxt);
+
 enum sdxi_submit_flags {
 	SDXI_CXT_SUBMIT_KICK  = BIT(0),
 	SDXI_CXT_SUBMIT_FENCE = BIT(1),

-- 
2.56.0

[PATCH v4 22/25] dmaengine: sdxi: Encode copy and interrupt descriptors

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Add copy and interrupt descriptor layouts and encoders. Both encoders
accept operation-specific parameter structs.

Copy descriptors are used to implement memcpy offload for the DMA
engine provider, and interrupt descriptors are used to signal the
completion of preceding descriptors in the ring.

Conditionally expose sdxi_encode_size32() for unit testing.

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/descriptor.c | 101 ++++++++++++++++++++++++++++++++++++++++++
 drivers/dma/sdxi/descriptor.h |  23 ++++++++++
 drivers/dma/sdxi/hw.h         |  27 +++++++++++
 3 files changed, 151 insertions(+)

diff --git a/drivers/dma/sdxi/descriptor.c b/drivers/dma/sdxi/descriptor.c
index 424453ff7ba0..25b5d2862e79 100644
--- a/drivers/dma/sdxi/descriptor.c
+++ b/drivers/dma/sdxi/descriptor.c
@@ -5,14 +5,115 @@
  * Copyright Advanced Micro Devices, Inc.
  */
 
+#include <kunit/test-bug.h>
 #include <kunit/visibility.h>
 #include <linux/bitfield.h>
+#include <linux/bug.h>
+#include <linux/range.h>
+#include <linux/sizes.h>
 #include <linux/types.h>
 #include <asm/byteorder.h>
 
 #include "hw.h"
 #include "descriptor.h"
 
+VISIBLE_IF_KUNIT int __must_check sdxi_encode_size32(u64 size, __le32 *dest)
+{
+	/*
+	 * Sizes are encoded as value - 1:
+	 * value    encoding
+	 *     1           0
+	 *     2           1
+	 *   ...
+	 *    4G  0xffffffff
+	 * See SDXI 1.0 Table 6-8: DSC_DMAB_COPY Descriptor Format,
+	 * for example.
+	 */
+	if (size > 0 && size <= SZ_4G) {
+		*dest = cpu_to_le32((u32)(size - 1));
+		return 0;
+	}
+
+	/* Inhibit warning stack traces while testing. */
+	if (!kunit_get_current_test()) {
+		WARN_ON_ONCE(size == 0);
+		WARN_ON_ONCE(size > SZ_4G);
+	}
+
+	return -EINVAL;
+}
+EXPORT_SYMBOL_IF_KUNIT(sdxi_encode_size32);
+
+int sdxi_encode_copy(struct sdxi_desc *desc, const struct sdxi_copy *params)
+{
+	u64 csb_ptr;
+	u32 opcode;
+	__le32 size;
+	int err;
+
+	err = sdxi_encode_size32(params->len, &size);
+	if (err)
+		return err;
+	/*
+	 * Reject overlapping src and dst. "Software ... shall not
+	 * overlap the source buffer, destination buffer, Atomic
+	 * Return Data, or completion status block." - SDXI 1.0 5.6
+	 * Memory Consistency Model
+	 */
+	if (range_overlaps(&(const struct range) {
+				   .start = params->src,
+				   .end   = params->src + params->len - 1,
+			   },
+			   &(const struct range) {
+				   .start = params->dst,
+				   .end   = params->dst + params->len - 1,
+			   }))
+		return -EINVAL;
+
+	opcode = (FIELD_PREP(SDXI_DSC_SUBTYPE, SDXI_DSC_OP_SUBTYPE_COPY) |
+		  FIELD_PREP(SDXI_DSC_TYPE, SDXI_DSC_OP_TYPE_DMAB));
+
+	csb_ptr = FIELD_PREP(SDXI_DSC_NP, 1);
+
+	*desc = (typeof(*desc)) {
+		.copy = (typeof(desc->copy)) {
+			.opcode = cpu_to_le32(opcode),
+			.size = size,
+			.akey0 = cpu_to_le16(params->src_akey),
+			.akey1 = cpu_to_le16(params->dst_akey),
+			.addr0 = cpu_to_le64(params->src),
+			.addr1 = cpu_to_le64(params->dst),
+			.csb_ptr = cpu_to_le64(csb_ptr),
+		},
+	};
+
+	return 0;
+}
+EXPORT_SYMBOL_IF_KUNIT(sdxi_encode_copy);
+
+int sdxi_encode_intr(struct sdxi_desc *desc,
+		     const struct sdxi_intr *params)
+{
+	u64 csb_ptr;
+	u32 opcode;
+
+	opcode = (FIELD_PREP(SDXI_DSC_SUBTYPE, SDXI_DSC_OP_SUBTYPE_INTR) |
+		  FIELD_PREP(SDXI_DSC_TYPE, SDXI_DSC_OP_TYPE_INTR));
+
+	csb_ptr = FIELD_PREP(SDXI_DSC_NP, 1);
+
+	*desc = (typeof(*desc)) {
+		.intr = (typeof(desc->intr)) {
+			.opcode = cpu_to_le32(opcode),
+			.akey = cpu_to_le16(params->akey),
+			.csb_ptr = cpu_to_le64(csb_ptr),
+		},
+	};
+
+	return 0;
+}
+EXPORT_SYMBOL_IF_KUNIT(sdxi_encode_intr);
+
 int sdxi_encode_cxt_start(struct sdxi_desc *desc,
 			  const struct sdxi_cxt_start *params)
 {
diff --git a/drivers/dma/sdxi/descriptor.h b/drivers/dma/sdxi/descriptor.h
index 9138f7c85f3a..cf071a281941 100644
--- a/drivers/dma/sdxi/descriptor.h
+++ b/drivers/dma/sdxi/descriptor.h
@@ -9,6 +9,7 @@
  */
 
 #include <linux/bitfield.h>
+#include <linux/kconfig.h>
 #include <linux/minmax.h>
 #include <linux/ratelimit.h>
 #include <linux/types.h>
@@ -16,6 +17,10 @@
 
 #include "hw.h"
 
+#if IS_ENABLED(CONFIG_KUNIT)
+int __must_check sdxi_encode_size32(u64 size, __le32 *dest);
+#endif
+
 static inline void sdxi_desc_vl_expect(const struct sdxi_desc *desc, bool expected)
 {
 	u8 vl = FIELD_GET(SDXI_DSC_VL, le32_to_cpu(desc->opcode));
@@ -80,6 +85,24 @@ static inline struct sdxi_cxt_range sdxi_cxt_range_single(u16 nr)
 	return sdxi_cxt_range(nr, nr);
 }
 
+struct sdxi_copy {
+	dma_addr_t src;
+	dma_addr_t dst;
+	u64 len;
+	u16 src_akey;
+	u16 dst_akey;
+};
+
+int sdxi_encode_copy(struct sdxi_desc *desc,
+		     const struct sdxi_copy *params);
+
+struct sdxi_intr {
+	u16 akey;
+};
+
+int sdxi_encode_intr(struct sdxi_desc *desc,
+		     const struct sdxi_intr *params);
+
 struct sdxi_cxt_start {
 	struct sdxi_cxt_range range;
 };
diff --git a/drivers/dma/sdxi/hw.h b/drivers/dma/sdxi/hw.h
index 51b2bcbeca90..71579eb3ac26 100644
--- a/drivers/dma/sdxi/hw.h
+++ b/drivers/dma/sdxi/hw.h
@@ -160,6 +160,25 @@ struct sdxi_desc {
 	static_assert(offsetof(struct tag_, csb_ptr) ==			\
 		      offsetof(struct sdxi_dsc_generic, csb_ptr))
 
+		/* SDXI 1.0 Table 6-8: DSC_DMAB_COPY Descriptor Format */
+		define_sdxi_dsc(sdxi_dsc_dmab_copy, copy,
+			__le32 size;
+			__u8 attr;
+			__u8 rsvd_0[3];
+			__le16 akey0;
+			__le16 akey1;
+			__le64 addr0;
+			__le64 addr1;
+			__u8 rsvd_1[24];
+		);
+
+		/* SDXI 1.0 Table 6-12: DSC_INTR Descriptor Format */
+		define_sdxi_dsc(sdxi_dsc_intr, intr,
+			__u8 rsvd_0[8];
+			__le16 akey;
+			__u8 rsvd_1[42];
+		);
+
 		/* SDXI 1.0 Table 6-14: DSC_CXT_START Descriptor Format */
 		define_sdxi_dsc(sdxi_dsc_cxt_start, cxt_start,
 			__u8 rsvd_0;
@@ -215,11 +234,19 @@ static_assert(sizeof(struct sdxi_desc) == 64);
 
 /* SDXI 1.0 Table 6-1: SDXI Operation Groups */
 enum sdxi_dsc_type {
+	SDXI_DSC_OP_TYPE_DMAB    = 0x001,
 	SDXI_DSC_OP_TYPE_ADMIN   = 0x002,
+	SDXI_DSC_OP_TYPE_INTR    = 0x004,
 };
 
 /* SDXI 1.0 Table 6-2: SDXI Operation Groups, Types, and Subtypes */
 enum sdxi_dsc_subtype {
+	/* DMA Base */
+	SDXI_DSC_OP_SUBTYPE_COPY    = 0x03,
+
+	/* Interrupt */
+	SDXI_DSC_OP_SUBTYPE_INTR = 0x00,
+
 	/* Administrative */
 	SDXI_DSC_OP_SUBTYPE_CXT_UPD      = 0x01,
 	SDXI_DSC_OP_SUBTYPE_CXT_START_NM = 0x03,

-- 
2.56.0

[PATCH v4 23/25] dmaengine: sdxi: Add unit tests for descriptor encoding

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Test the encoder function for each descriptor type the driver encodes.

Decode descriptors with the packing API using independently specified
bit offsets, so errors in the production bitfield masks can be detected.

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/Kconfig            |   1 +
 drivers/dma/sdxi/Makefile           |   1 +
 drivers/dma/sdxi/descriptor_kunit.c | 537 ++++++++++++++++++++++++++++++++++++
 3 files changed, 539 insertions(+)

diff --git a/drivers/dma/sdxi/Kconfig b/drivers/dma/sdxi/Kconfig
index 7217375c9216..314b3e725ccb 100644
--- a/drivers/dma/sdxi/Kconfig
+++ b/drivers/dma/sdxi/Kconfig
@@ -31,6 +31,7 @@ config SDXI_KUNIT_TEST
 	tristate "SDXI unit tests" if !KUNIT_ALL_TESTS
 	depends on SDXI_CORE && KUNIT
 	default KUNIT_ALL_TESTS
+	select PACKING
 	help
 	  KUnit tests for parts of the SDXI driver. Does not require
 	  SDXI hardware.
diff --git a/drivers/dma/sdxi/Makefile b/drivers/dma/sdxi/Makefile
index e9c180ea6fa4..1e261fb03225 100644
--- a/drivers/dma/sdxi/Makefile
+++ b/drivers/dma/sdxi/Makefile
@@ -12,4 +12,5 @@ sdxi-pci-y := pci.o
 
 obj-$(CONFIG_SDXI_KUNIT_TEST) += \
 	completion_kunit.o    \
+	descriptor_kunit.o    \
 	ring_kunit.o
diff --git a/drivers/dma/sdxi/descriptor_kunit.c b/drivers/dma/sdxi/descriptor_kunit.c
new file mode 100644
index 000000000000..fdca024e3e31
--- /dev/null
+++ b/drivers/dma/sdxi/descriptor_kunit.c
@@ -0,0 +1,537 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * SDXI descriptor encoding tests.
+ *
+ * Copyright Advanced Micro Devices, Inc.
+ *
+ * Decode descriptors using packing layouts specified independently of the
+ * production bitfield masks, allowing the tests to detect incorrect field
+ * definitions.
+ *
+ * Defining SKIP_PACKING_CHECKS bypasses compile-time packing-layout validation
+ * to reduce build time.
+ */
+#include <kunit/device.h>
+#include <kunit/test-bug.h>
+#include <kunit/test.h>
+#include <linux/container_of.h>
+#include <linux/dma-mapping.h>
+#include <linux/module.h>
+#include <linux/packing.h>
+#include <linux/stddef.h>
+#include <linux/string.h>
+
+#include "descriptor.h"
+
+/* #define SKIP_PACKING_CHECKS */
+
+MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
+
+enum {
+	SDXI_PACKING_QUIRKS = QUIRK_LITTLE_ENDIAN | QUIRK_LSW32_IS_FIRST,
+};
+
+
+#define desc_field(_high, _low, _target_struct, _member) \
+	PACKED_FIELD(_high, _low, _target_struct, _member)
+#define desc_flag(_bit, _target_struct, _member) \
+	desc_field(_bit, _bit, _target_struct, _member)
+
+/* DMAB_COPY */
+struct unpacked__copy {
+	u32 size;
+	u8 attr_src;
+	u8 attr_dst;
+	u16 akey0;
+	u16 akey1;
+	u64 addr0;
+	u64 addr1;
+};
+
+#define copy_field(_high, _low, _member) \
+	desc_field(_high, _low, struct unpacked__copy, _member)
+
+static const struct packed_field_u16 copy_subfields[] = {
+	copy_field(63, 32, size),
+	copy_field(67, 64, attr_src),
+	copy_field(71, 68, attr_dst),
+	copy_field(111, 96, akey0),
+	copy_field(127, 112, akey1),
+	copy_field(191, 128, addr0),
+	copy_field(255, 192, addr1),
+};
+
+/* DSC_INTR */
+struct unpacked__intr {
+	u16 akey;
+};
+
+#define intr_field(_high, _low, _member) \
+	desc_field(_high, _low, struct unpacked__intr, _member)
+
+static const struct packed_field_u16 intr_subfields[] = {
+	intr_field(111, 96, akey),
+};
+
+/* DSC_SYNC */
+struct unpacked__sync {
+	u8 flt;
+	bool vf;
+	u16 vf_num;
+	u16 cxt_start;
+	u16 cxt_end;
+	u16 key_start;
+	u16 key_end;
+};
+
+#define sync_field(_high, _low, _member) \
+	desc_field(_high, _low, struct unpacked__sync, _member)
+#define sync_flag(_bit, _member) sync_field(_bit, _bit, _member)
+
+static const struct packed_field_u16 sync_subfields[] = {
+	sync_field(34, 32, flt),
+	sync_flag(47, vf),
+	sync_field(63, 48, vf_num),
+	sync_field(79, 64, cxt_start),
+	sync_field(95, 80, cxt_end),
+	sync_field(111, 96, key_start),
+	sync_field(127, 112, key_end),
+};
+
+/* DSC_CXT_START */
+struct unpacked__cxt_start {
+	bool dv;
+	bool vf;
+	u16 vf_num;
+	u16 cxt_start;
+	u16 cxt_end;
+	u64 db_value;
+};
+
+#define cxt_start_field(_high, _low, _member) \
+	desc_field(_high, _low, struct unpacked__cxt_start, _member)
+#define cxt_start_flag(_bit, _member) cxt_start_field(_bit, _bit, _member)
+
+static const struct packed_field_u16 cxt_start_subfields[] = {
+	cxt_start_flag(46, dv),
+	cxt_start_flag(47, vf),
+	cxt_start_field(63, 48, vf_num),
+	cxt_start_field(79, 64, cxt_start),
+	cxt_start_field(95, 80, cxt_end),
+	cxt_start_field(191, 128, db_value),
+};
+
+/* DSC_CXT_STOP */
+struct unpacked__cxt_stop {
+	bool hs;
+	bool vf;
+	u16 vf_num;
+	u16 cxt_start;
+	u16 cxt_end;
+};
+
+#define cxt_stop_field(_high, _low, _member) \
+	desc_field(_high, _low, struct unpacked__cxt_stop, _member)
+#define cxt_stop_flag(_bit, _member) cxt_stop_field(_bit, _bit, _member)
+
+static const struct packed_field_u16 cxt_stop_subfields[] = {
+	cxt_stop_flag(45, hs),
+	cxt_stop_flag(47, vf),
+	cxt_stop_field(63, 48, vf_num),
+	cxt_stop_field(79, 64, cxt_start),
+	cxt_stop_field(95, 80, cxt_end),
+};
+
+/* DSC_CXT_UPD */
+struct unpacked__cxt_upd {
+	u8 pcd;
+	bool vf;
+	u16 vf_num;
+	u16 cxt_start;
+	u16 cxt_end;
+};
+
+#define cxt_upd_field(_high, _low, _member) \
+	desc_field(_high, _low, struct unpacked__cxt_upd, _member)
+#define cxt_upd_flag(_bit, _member) cxt_upd_field(_bit, _bit, _member)
+
+static const struct packed_field_u16 cxt_upd_subfields[] = {
+	cxt_upd_field(34, 32, pcd),
+	cxt_upd_flag(47, vf),
+	cxt_upd_field(63, 48, vf_num),
+	cxt_upd_field(79, 64, cxt_start),
+	cxt_upd_field(95, 80, cxt_end),
+};
+
+/* DSC_GENERIC */
+struct unpacked_desc {
+	u64 csb_ptr;
+	u16 type;
+	u8 subtype;
+	bool vl;
+	bool se;
+	bool fe;
+	bool ch;
+	bool csr;
+	bool rb;
+	bool np;
+	union {
+		struct unpacked__copy copy;
+		struct unpacked__intr intr;
+		struct unpacked__sync sync;
+		struct unpacked__cxt_start cxt_start;
+		struct unpacked__cxt_stop cxt_stop;
+		struct unpacked__cxt_upd cxt_upd;
+	};
+};
+
+#define generic_field(_high, _low, _member)			\
+	desc_field(_high, _low, struct unpacked_desc, _member)
+#define generic_flag(_bit, _member) generic_field(_bit, _bit, _member)
+
+static const struct packed_field_u16 generic_subfields[] = {
+	generic_flag(0, vl),
+	generic_flag(1, se),
+	generic_flag(2, fe),
+	generic_flag(3, ch),
+	generic_flag(4, csr),
+	generic_flag(5, rb),
+	generic_field(15, 8, subtype),
+	generic_field(26, 16, type),
+	generic_flag(448, np),
+	generic_field(511, 453, csb_ptr),
+};
+
+#ifndef SKIP_PACKING_CHECKS
+#define define_unpack_fn(_T)						\
+	static void unpack_ ## _T(struct unpacked_desc *to,		\
+				  const struct sdxi_desc *from)		\
+	{								\
+		unpack_fields(from, sizeof(*from), to,	\
+			      generic_subfields, SDXI_PACKING_QUIRKS);	\
+		unpack_fields(from, sizeof(*from), &to->_T,		\
+			      _T ## _subfields, SDXI_PACKING_QUIRKS);	\
+	}
+#else
+#define define_unpack_fn(_T)						\
+	static void unpack_ ## _T(struct unpacked_desc *to,		\
+				  const struct sdxi_desc *from)		\
+	{								\
+		unpack_fields_u16(from, sizeof(*from), to,		\
+				  generic_subfields,			\
+				  ARRAY_SIZE(generic_subfields),	\
+				  SDXI_PACKING_QUIRKS);			\
+		unpack_fields_u16(from, sizeof(*from), &to->_T,		\
+				  _T ## _subfields,			\
+				  ARRAY_SIZE(_T ## _subfields),		\
+				  SDXI_PACKING_QUIRKS);			\
+	}
+#endif	/* SKIP_PACKING_CHECKS */
+
+define_unpack_fn(intr)
+define_unpack_fn(copy)
+define_unpack_fn(sync)
+define_unpack_fn(cxt_start)
+define_unpack_fn(cxt_stop)
+define_unpack_fn(cxt_upd)
+
+static void desc_poison(struct sdxi_desc *d)
+{
+	memset(d, 0xff, sizeof(*d));
+}
+
+static void encode_size32(struct kunit *t)
+{
+	__le32 res = cpu_to_le32(U32_MAX);
+
+	/* Valid sizes. */
+	KUNIT_EXPECT_EQ(t, 0, sdxi_encode_size32(1, &res));
+	KUNIT_EXPECT_EQ(t, 0, le32_to_cpu(res));
+
+	KUNIT_EXPECT_EQ(t, 0, sdxi_encode_size32(SZ_4K, &res));
+	KUNIT_EXPECT_EQ(t, SZ_4K - 1, le32_to_cpu(res));
+
+	KUNIT_EXPECT_EQ(t, 0, sdxi_encode_size32(SZ_4M, &res));
+	KUNIT_EXPECT_EQ(t, SZ_4M - 1, le32_to_cpu(res));
+
+	KUNIT_EXPECT_EQ(t, 0, sdxi_encode_size32(SZ_4G - 1, &res));
+	KUNIT_EXPECT_EQ(t, SZ_4G - 2, le32_to_cpu(res));
+
+	KUNIT_EXPECT_EQ(t, 0, sdxi_encode_size32(SZ_4G, &res));
+	KUNIT_EXPECT_EQ(t, SZ_4G - 1, le32_to_cpu(res));
+
+	/* Invalid sizes. Ensure the out parameter is unmodified. */
+#define RES_VAL 0x843829
+	res = cpu_to_le32(RES_VAL);
+
+	KUNIT_EXPECT_EQ(t, -EINVAL, sdxi_encode_size32(0, &res));
+	KUNIT_EXPECT_EQ(t, RES_VAL, le32_to_cpu(res));
+
+	KUNIT_EXPECT_EQ(t, -EINVAL, sdxi_encode_size32(SZ_4G + 1, &res));
+	KUNIT_EXPECT_EQ(t, RES_VAL, le32_to_cpu(res));
+
+	KUNIT_EXPECT_EQ(t, -EINVAL, sdxi_encode_size32(SZ_8G, &res));
+	KUNIT_EXPECT_EQ(t, RES_VAL, le32_to_cpu(res));
+
+	KUNIT_EXPECT_EQ(t, -EINVAL, sdxi_encode_size32(U64_MAX, &res));
+	KUNIT_EXPECT_EQ(t, RES_VAL, le32_to_cpu(res));
+
+#undef RES_VAL
+}
+
+static void copy(struct kunit *t)
+{
+	struct unpacked_desc unpacked;
+	struct sdxi_desc desc = {};
+	struct sdxi_copy copy = {
+		.src = 0x1000,
+		.dst = 0x2000,
+		.len = 4096,
+		.src_akey = 0,
+		.dst_akey = 0,
+	};
+
+	KUNIT_EXPECT_EQ(t, 0, sdxi_encode_copy(&desc, &copy));
+
+	unpack_copy(&unpacked, &desc);
+	KUNIT_EXPECT_EQ(t, unpacked.vl, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.ch, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.subtype, SDXI_DSC_OP_SUBTYPE_COPY);
+	KUNIT_EXPECT_EQ(t, unpacked.type, SDXI_DSC_OP_TYPE_DMAB);
+	KUNIT_EXPECT_EQ(t, unpacked.csb_ptr, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.np, 1);
+
+	KUNIT_EXPECT_EQ(t, unpacked.copy.size, copy.len - 1);
+
+	/* Size boundaries. */
+	desc_poison(&desc);
+	copy.len = 0;
+	KUNIT_EXPECT_EQ(t, -EINVAL, sdxi_encode_copy(&desc, &copy));
+
+	desc_poison(&desc);
+	copy.len = 1;
+	KUNIT_EXPECT_EQ(t, 0, sdxi_encode_copy(&desc, &copy));
+	unpack_copy(&unpacked, &desc);
+	KUNIT_EXPECT_EQ(t, unpacked.copy.size, copy.len - 1);
+
+	/* SDXI forbids overlapping source and destination. */
+	desc_poison(&desc);
+	copy.len = 4097;
+	KUNIT_EXPECT_EQ(t, -EINVAL, sdxi_encode_copy(&desc, &copy));
+	copy = (typeof(copy)) {
+		.src = 0x4000,
+		.dst = 0x4000,
+		.len = 1,
+		.src_akey = 0,
+		.dst_akey = 0,
+	};
+	KUNIT_EXPECT_EQ(t, -EINVAL, sdxi_encode_copy(&desc, &copy));
+
+	desc_poison(&desc);
+	KUNIT_EXPECT_EQ(t, 0,
+			sdxi_encode_copy(&desc,
+					 &(struct sdxi_copy) {
+						 .src = 0x1000,
+						 .dst = 0x2000,
+						 .len = 0x100,
+						 .src_akey = 1,
+						 .dst_akey = 2,
+					 }));
+	KUNIT_EXPECT_EQ(t, 0x1000, le64_to_cpu(desc.copy.addr0));
+	KUNIT_EXPECT_EQ(t, 0x2000, le64_to_cpu(desc.copy.addr1));
+	KUNIT_EXPECT_EQ(t, 0x100, 1 + le32_to_cpu(desc.copy.size));
+	KUNIT_EXPECT_EQ(t, 1, le16_to_cpu(desc.copy.akey0));
+	KUNIT_EXPECT_EQ(t, 2, le16_to_cpu(desc.copy.akey1));
+
+	unpack_copy(&unpacked, &desc);
+	KUNIT_EXPECT_EQ(t, unpacked.vl, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.ch, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.subtype, SDXI_DSC_OP_SUBTYPE_COPY);
+	KUNIT_EXPECT_EQ(t, unpacked.type, SDXI_DSC_OP_TYPE_DMAB);
+	KUNIT_EXPECT_EQ(t, unpacked.csb_ptr, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.np, 1);
+
+	KUNIT_EXPECT_EQ(t, unpacked.copy.size, 0x100 - 1);
+}
+
+static void intr(struct kunit *t)
+{
+	struct unpacked_desc unpacked;
+	struct sdxi_intr intr = {
+		.akey = 1234,
+	};
+	struct sdxi_desc desc;
+
+	desc_poison(&desc);
+	KUNIT_EXPECT_EQ(t, 0, sdxi_encode_intr(&desc, &intr));
+
+	unpack_intr(&unpacked, &desc);
+	KUNIT_EXPECT_EQ(t, unpacked.vl, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.ch, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.subtype, SDXI_DSC_OP_SUBTYPE_INTR);
+	KUNIT_EXPECT_EQ(t, unpacked.type, SDXI_DSC_OP_TYPE_INTR);
+	KUNIT_EXPECT_EQ(t, unpacked.csb_ptr, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.np, 1);
+
+	KUNIT_EXPECT_EQ(t, unpacked.intr.akey, 1234);
+}
+
+static void cxt_start(struct kunit *t)
+{
+	struct unpacked_desc unpacked;
+	struct sdxi_cxt_start start = {
+		.range = sdxi_cxt_range_single(2),
+	};
+	struct sdxi_desc desc;
+
+	desc_poison(&desc);
+	KUNIT_ASSERT_EQ(t, 0, sdxi_encode_cxt_start(&desc, &start));
+
+	unpack_cxt_start(&unpacked, &desc);
+
+	KUNIT_EXPECT_EQ(t, 0, desc.cxt_start.vflags);
+
+	KUNIT_EXPECT_EQ(t, unpacked.vl, 0);
+	/* Context start requires FE=1 and SE=0. */
+	KUNIT_EXPECT_EQ(t, unpacked.se, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.fe, 1);
+	KUNIT_EXPECT_EQ(t, unpacked.ch, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.subtype, SDXI_DSC_OP_SUBTYPE_CXT_START_NM);
+	KUNIT_EXPECT_EQ(t, unpacked.type, SDXI_DSC_OP_TYPE_ADMIN);
+	KUNIT_EXPECT_EQ(t, unpacked.csb_ptr, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.np, 1);
+
+	KUNIT_EXPECT_FALSE(t, unpacked.cxt_start.dv);
+	KUNIT_EXPECT_FALSE(t, unpacked.cxt_start.vf);
+	KUNIT_EXPECT_EQ(t, unpacked.cxt_start.cxt_start, 2);
+	KUNIT_EXPECT_EQ(t, unpacked.cxt_start.cxt_end, 2);
+	KUNIT_EXPECT_EQ(t, unpacked.cxt_start.vf_num, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.cxt_start.db_value, 0);
+}
+
+static void cxt_stop(struct kunit *t)
+{
+	struct unpacked_desc unpacked;
+	struct sdxi_cxt_stop stop = {
+		.range = sdxi_cxt_range_single(2),
+	};
+	struct sdxi_desc desc;
+
+	desc_poison(&desc);
+	KUNIT_ASSERT_EQ(t, 0, sdxi_encode_cxt_stop(&desc, &stop));
+
+	unpack_cxt_stop(&unpacked, &desc);
+
+	KUNIT_EXPECT_EQ(t, 0, desc.cxt_stop.vflags);
+
+	KUNIT_EXPECT_EQ(t, unpacked.vl, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.se, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.fe, 1);
+	KUNIT_EXPECT_EQ(t, unpacked.ch, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.subtype, SDXI_DSC_OP_SUBTYPE_CXT_STOP);
+	KUNIT_EXPECT_EQ(t, unpacked.type, SDXI_DSC_OP_TYPE_ADMIN);
+	KUNIT_EXPECT_EQ(t, unpacked.csb_ptr, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.np, 1);
+
+	KUNIT_EXPECT_FALSE(t, unpacked.cxt_stop.hs);
+	KUNIT_EXPECT_FALSE(t, unpacked.cxt_stop.vf);
+	KUNIT_EXPECT_EQ(t, unpacked.cxt_stop.cxt_start, 2);
+	KUNIT_EXPECT_EQ(t, unpacked.cxt_stop.cxt_end, 2);
+	KUNIT_EXPECT_EQ(t, unpacked.cxt_stop.vf_num, 0);
+}
+
+static void cxt_upd(struct kunit *t)
+{
+	struct unpacked_desc unpacked;
+	struct sdxi_cxt_upd upd = {
+		.level = SDXI_CXT_UPD_LVL_L2,
+		.range = sdxi_cxt_range(128, 255),
+	};
+	struct sdxi_desc desc;
+
+	desc_poison(&desc);
+	KUNIT_ASSERT_EQ(t, 0, sdxi_encode_cxt_upd(&desc, &upd));
+
+	unpack_cxt_upd(&unpacked, &desc);
+
+	KUNIT_EXPECT_EQ(t, 0, desc.cxt_upd.vflags);
+
+	KUNIT_EXPECT_EQ(t, unpacked.vl, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.se, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.ch, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.subtype, SDXI_DSC_OP_SUBTYPE_CXT_UPD);
+	KUNIT_EXPECT_EQ(t, unpacked.type, SDXI_DSC_OP_TYPE_ADMIN);
+	KUNIT_EXPECT_EQ(t, unpacked.csb_ptr, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.np, 1);
+
+	KUNIT_EXPECT_EQ(t, unpacked.cxt_upd.pcd, SDXI_CXT_UPD_LVL_L2);
+	KUNIT_EXPECT_FALSE(t, unpacked.cxt_upd.vf);
+	KUNIT_EXPECT_EQ(t, unpacked.cxt_upd.vf_num, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.cxt_upd.cxt_start, 128);
+	KUNIT_EXPECT_EQ(t, unpacked.cxt_upd.cxt_end, 255);
+
+	/* The other defined levels encode into the tgt_pcd subfield. */
+	desc_poison(&desc);
+	upd.level = SDXI_CXT_UPD_LVL_CTL;
+	KUNIT_ASSERT_EQ(t, 0, sdxi_encode_cxt_upd(&desc, &upd));
+	unpack_cxt_upd(&unpacked, &desc);
+	KUNIT_EXPECT_EQ(t, unpacked.cxt_upd.pcd, SDXI_CXT_UPD_LVL_CTL);
+
+	desc_poison(&desc);
+	upd.level = SDXI_CXT_UPD_LVL_L1;
+	KUNIT_ASSERT_EQ(t, 0, sdxi_encode_cxt_upd(&desc, &upd));
+	unpack_cxt_upd(&unpacked, &desc);
+	KUNIT_EXPECT_EQ(t, unpacked.cxt_upd.pcd, SDXI_CXT_UPD_LVL_L1);
+}
+
+static void sync(struct kunit *t)
+{
+	struct sdxi_sync sync = {
+		.filter = SDXI_SYNC_FLT_STOP,
+		.range = sdxi_cxt_range(1, U16_MAX),
+	};
+	struct sdxi_desc desc;
+	struct unpacked_desc unpacked;
+
+	desc_poison(&desc);
+	KUNIT_ASSERT_EQ(t, 0, sdxi_encode_sync(&desc, &sync));
+	unpack_sync(&unpacked, &desc);
+
+	KUNIT_EXPECT_EQ(t, unpacked.type, SDXI_DSC_OP_TYPE_ADMIN);
+	KUNIT_EXPECT_EQ(t, unpacked.subtype, SDXI_DSC_OP_SUBTYPE_SYNC);
+	KUNIT_EXPECT_EQ(t, unpacked.ch, 0);
+	KUNIT_EXPECT_EQ(t, unpacked.sync.flt, SDXI_SYNC_FLT_STOP);
+	KUNIT_EXPECT_EQ(t, unpacked.sync.cxt_start, 1);
+	KUNIT_EXPECT_EQ(t, unpacked.sync.cxt_end, U16_MAX);
+}
+
+static struct kunit_case generic_desc_tcs[] = {
+	KUNIT_CASE(encode_size32),
+	KUNIT_CASE(copy),
+	KUNIT_CASE(intr),
+	KUNIT_CASE(cxt_start),
+	KUNIT_CASE(cxt_stop),
+	KUNIT_CASE(cxt_upd),
+	KUNIT_CASE(sync),
+	{}
+};
+
+static int generic_desc_setup_device(struct kunit *t)
+{
+	struct device *dev = kunit_device_register(t, "sdxi-mock-device");
+
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(t, dev);
+	t->priv = dev;
+	return 0;
+}
+
+static struct kunit_suite generic_desc_ts = {
+	.name = "Generic SDXI descriptor encoding",
+	.test_cases = generic_desc_tcs,
+	.init = generic_desc_setup_device,
+};
+kunit_test_suite(generic_desc_ts);
+
+MODULE_DESCRIPTION("SDXI descriptor encoding tests");
+MODULE_AUTHOR("Nathan Lynch");
+MODULE_LICENSE("GPL");

-- 
2.56.0

[PATCH v4 24/25] dmaengine: sdxi: MSI/MSI-X vector allocation and mapping

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Request MSI/MSI-X vectors for the error log interrupt and up to one
interrupt per supported client context. Reserve vector 0 for the error log
and track the remaining allocated vectors with an IDA.

Introduce sdxi_alloc_vector() and sdxi_free_vector() which are thin
wrappers around the IDA that tracks the allocated vector range.

Introduce sdxi_vector_to_irq() which invokes a new get_irq() bus op to
translate the device-relative index to the Linux IRQ number for use
with request_irq() etc. For PCI this dispatches to pci_irq_vector().

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/device.c |  3 +++
 drivers/dma/sdxi/pci.c    | 30 ++++++++++++++++++++++++++-
 drivers/dma/sdxi/sdxi.h   | 53 +++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 85 insertions(+), 1 deletion(-)

diff --git a/drivers/dma/sdxi/device.c b/drivers/dma/sdxi/device.c
index 8786be71d3e3..30c4e687cedf 100644
--- a/drivers/dma/sdxi/device.c
+++ b/drivers/dma/sdxi/device.c
@@ -11,6 +11,7 @@
 #include <linux/dma-mapping.h>
 #include <linux/dmapool.h>
 #include <linux/export.h>
+#include <linux/idr.h>
 #include <linux/iopoll.h>
 #include <linux/jiffies.h>
 #include <linux/limits.h>
@@ -437,6 +438,7 @@ int sdxi_register(struct device *dev, const struct sdxi_bus_ops *ops)
 
 	sdxi->dev = dev;
 	sdxi->bus_ops = ops;
+	ida_init(&sdxi->vectors);
 	xa_init_flags(&sdxi->client_cxts, XA_FLAGS_ALLOC1);
 	dev_set_drvdata(dev, sdxi);
 
@@ -459,6 +461,7 @@ void sdxi_unregister(struct device *dev)
 	/* Stop/reset is best effort if the hardware cannot be quiesced. */
 	sdxi_dev_stop(sdxi);
 	sdxi_client_cxts_exit(sdxi);
+	ida_destroy(&sdxi->vectors);
 }
 EXPORT_SYMBOL_NS_GPL(sdxi_unregister, "SDXI");
 
diff --git a/drivers/dma/sdxi/pci.c b/drivers/dma/sdxi/pci.c
index d196ab5cf6f1..ef978ca22db1 100644
--- a/drivers/dma/sdxi/pci.c
+++ b/drivers/dma/sdxi/pci.c
@@ -31,8 +31,9 @@ static int sdxi_pci_init(struct sdxi_dev *sdxi)
 {
 	struct pci_dev *pdev = sdxi_to_pci_dev(sdxi);
 	struct device *dev = &pdev->dev;
+	u32 cap1_max_cxt;
 	u64 cap1;
-	int ret;
+	int vecs, ret;
 
 	ret = pcim_enable_device(pdev);
 	if (ret)
@@ -63,6 +64,27 @@ static int sdxi_pci_init(struct sdxi_dev *sdxi)
 	sdxi->has_mailbox = pdev->is_virtfn ||
 		pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_SRIOV);
 
+	/*
+	 * Allocate the minimum required set of vectors plus one for
+	 * each client context supported by the function.
+	 *
+	 * pci_alloc_irq_vectors() handles calling pci_free_irq_vectors()
+	 * automatically despite not being called pcim_*. See
+	 * pci_setup_msi_context().
+	 */
+	cap1_max_cxt = FIELD_GET(SDXI_MMIO_CAP1_MAX_CXT,
+				 sdxi_read64(sdxi, SDXI_MMIO_CAP1));
+	vecs = pci_alloc_irq_vectors(pdev, SDXI_MIN_VECTORS,
+				     SDXI_MIN_VECTORS + cap1_max_cxt,
+				     PCI_IRQ_MSI | PCI_IRQ_MSIX);
+	if (vecs < 0)
+		return dev_err_probe(dev, vecs,
+				     "failed to allocate MSIs (max_cxt=%u)\n",
+				     cap1_max_cxt);
+
+	sdxi->nr_vectors = vecs;
+	dev_dbg(sdxi->dev, "allocated %u vectors\n", sdxi->nr_vectors);
+
 	pci_set_master(pdev);
 	return 0;
 }
@@ -86,9 +108,15 @@ static int sdxi_pci_reset(struct sdxi_dev *sdxi)
 	return err;
 }
 
+static int sdxi_pci_get_irq(struct sdxi_dev *sdxi, unsigned int nr)
+{
+	return pci_irq_vector(sdxi_to_pci_dev(sdxi), nr);
+}
+
 static const struct sdxi_bus_ops sdxi_pci_ops = {
 	.init = sdxi_pci_init,
 	.reset = sdxi_pci_reset,
+	.get_irq = sdxi_pci_get_irq,
 };
 
 static int sdxi_pci_probe(struct pci_dev *pdev,
diff --git a/drivers/dma/sdxi/sdxi.h b/drivers/dma/sdxi/sdxi.h
index aaa43eab00b3..07044d837d9a 100644
--- a/drivers/dma/sdxi/sdxi.h
+++ b/drivers/dma/sdxi/sdxi.h
@@ -8,8 +8,10 @@
 #ifndef DMA_SDXI_H
 #define DMA_SDXI_H
 
+#include <linux/bug.h>
 #include <linux/compiler_types.h>
 #include <linux/dev_printk.h>
+#include <linux/idr.h>
 #include <linux/io.h>
 #include <linux/types.h>
 #include <linux/xarray.h>
@@ -25,6 +27,18 @@
 #define L1_CXT_CTRL_PTR_SHIFT		6
 #define L1_CXT_AKEY_PTR_SHIFT		12
 
+enum {
+	/* Reserve vector 0 for the error log interrupt. */
+	SDXI_ERROR_VECTOR = 0,
+
+	/*
+	 * Request at least one vector to account for the error log
+	 * interrupt. Increment this if the driver gains more
+	 * dedicated interrupts (e.g. one for the admin context).
+	 */
+	SDXI_MIN_VECTORS = 1,
+};
+
 struct sdxi_dev;
 
 /**
@@ -41,6 +55,10 @@ struct sdxi_bus_ops {
 	 * @reset: Bus-specific recovery when the native stop sequence fails.
 	 */
 	int (*reset)(struct sdxi_dev *sdxi);
+	/**
+	 * @get_irq: Map device interrupt index to Linux IRQ number.
+	 */
+	int (*get_irq)(struct sdxi_dev *sdxi, unsigned int index);
 };
 
 struct sdxi_dev {
@@ -61,6 +79,9 @@ struct sdxi_dev {
 
 	struct dma_pool *cxt_ctl_pool;
 
+	unsigned int nr_vectors;
+	struct ida vectors;
+
 	struct sdxi_cxt *admin_cxt;
 	struct xarray client_cxts; /* context id -> (struct sdxi_cxt *) */
 
@@ -69,6 +90,38 @@ struct sdxi_dev {
 	const struct sdxi_bus_ops *bus_ops;
 };
 
+/**
+ * sdxi_alloc_vector() - Allocate an interrupt vector.
+ *
+ * A vector that will have the same lifetime as the device does not
+ * need to be released explicitly. Otherwise the vector must be
+ * released with sdxi_free_vector().
+ */
+static inline int sdxi_alloc_vector(struct sdxi_dev *sdxi)
+{
+	return ida_alloc_range(&sdxi->vectors, SDXI_MIN_VECTORS,
+			       sdxi->nr_vectors - 1, GFP_KERNEL);
+}
+
+/**
+ * sdxi_free_vector() - Release a previously allocated index.
+ */
+static inline void sdxi_free_vector(struct sdxi_dev *sdxi, unsigned int nr)
+{
+	ida_free(&sdxi->vectors, nr);
+}
+
+/**
+ * sdxi_vector_to_irq() - Translate an allocated interrupt vector to
+ *                        Linux IRQ number suitable for passing to
+ *                        request_irq() et al.
+ */
+static inline int sdxi_vector_to_irq(struct sdxi_dev *sdxi, unsigned int nr)
+{
+	WARN_ON_ONCE(!ida_exists(&sdxi->vectors, nr));
+	return sdxi->bus_ops->get_irq(sdxi, nr);
+}
+
 int sdxi_register(struct device *dev, const struct sdxi_bus_ops *ops);
 void sdxi_unregister(struct device *dev);
 

-- 
2.56.0

[PATCH v4 25/25] dmaengine: sdxi: Add DMA engine provider

[Nathan Lynch via B4 Relay]

From: Nathan Lynch <nathan.lynch@amd.com>

Register a virt-dma memcpy provider with a configurable number of channels
per function. Publish transactions in cookie order, batching copies behind
fenced completion interrupts. Keep transactions pending when ring slots or
completion blocks are unavailable.

Published descriptors cannot be withdrawn. Termination frees submitted and
pending transactions and retains published transactions for reaping without
completing cookies or invoking callbacks.

Synchronize by polling and reaping transactions under the channel lock,
then flushing virt-dma callbacks. The wait is uninterruptible and bounded
to ten seconds. On timeout, suppress remaining callbacks and retain unreaped
status blocks until context teardown; this timeout path does not establish
hardware quiescence for client buffers.

Co-developed-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Wei Huang <wei.huang2@amd.com>
Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>
---
 drivers/dma/sdxi/Kconfig  |   1 +
 drivers/dma/sdxi/Makefile |   1 +
 drivers/dma/sdxi/device.c |   2 +
 drivers/dma/sdxi/dma.c    | 509 ++++++++++++++++++++++++++++++++++++++++++++++
 drivers/dma/sdxi/dma.h    |  11 +
 5 files changed, 524 insertions(+)

diff --git a/drivers/dma/sdxi/Kconfig b/drivers/dma/sdxi/Kconfig
index 314b3e725ccb..85e23608405c 100644
--- a/drivers/dma/sdxi/Kconfig
+++ b/drivers/dma/sdxi/Kconfig
@@ -4,6 +4,7 @@ config SDXI_CORE
 	# SDXI 1.0 9 MMIO Control Registers.
 	depends on 64BIT
 	select DMA_ENGINE
+	select DMA_VIRTUAL_CHANNELS
 	help
 	  Enable support for Smart Data Accelerator Interface (SDXI)
 	  Platform Data Mover devices. SDXI is a vendor-neutral
diff --git a/drivers/dma/sdxi/Makefile b/drivers/dma/sdxi/Makefile
index 1e261fb03225..3a1d36997b44 100644
--- a/drivers/dma/sdxi/Makefile
+++ b/drivers/dma/sdxi/Makefile
@@ -5,6 +5,7 @@ sdxi-core-y := \
 	context.o     \
 	descriptor.o  \
 	device.o      \
+	dma.o         \
 	ring.o
 
 obj-$(CONFIG_SDXI_PCI) += sdxi-pci.o
diff --git a/drivers/dma/sdxi/device.c b/drivers/dma/sdxi/device.c
index 30c4e687cedf..265ebea1e350 100644
--- a/drivers/dma/sdxi/device.c
+++ b/drivers/dma/sdxi/device.c
@@ -23,6 +23,7 @@
 #include <linux/xarray.h>
 
 #include "context.h"
+#include "dma.h"
 #include "hw.h"
 #include "mmio.h"
 #include "sdxi.h"
@@ -424,6 +425,7 @@ static int sdxi_device_init(struct sdxi_dev *sdxi)
 	if (err)
 		return err;
 
+	sdxi_dma_register(sdxi);
 	return 0;
 }
 
diff --git a/drivers/dma/sdxi/dma.c b/drivers/dma/sdxi/dma.c
new file mode 100644
index 000000000000..512855c83da2
--- /dev/null
+++ b/drivers/dma/sdxi/dma.c
@@ -0,0 +1,509 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * SDXI dmaengine provider
+ *
+ * Copyright Advanced Micro Devices, Inc.
+ */
+
+#include <linux/cleanup.h>
+#include <linux/delay.h>
+#include <linux/dev_printk.h>
+#include <linux/container_of.h>
+#include <linux/dma-mapping.h>
+#include <linux/dmaengine.h>
+#include <linux/iopoll.h>
+#include <linux/list.h>
+#include <linux/lockdep.h>
+#include <linux/module.h>
+#include <linux/overflow.h>
+#include <linux/spinlock.h>
+
+#include "../dmaengine.h"
+#include "../virt-dma.h"
+#include "completion.h"
+#include "context.h"
+#include "descriptor.h"
+#include "dma.h"
+#include "ring.h"
+#include "sdxi.h"
+
+static unsigned short dma_channels = 1;
+module_param(dma_channels, ushort, 0644);
+MODULE_PARM_DESC(dma_channels, "DMA channels per function (default: 1)");
+
+/*
+ * Each channel owns an SDXI context. Transactions enter the ring in cookie
+ * order. Each transaction produces one copy descriptor with its own completion
+ * block, and batches end with a shared fenced interrupt descriptor. Each copy
+ * owns its completion block until reaped. Transactions awaiting capacity remain
+ * pending. Termination retains published transactions for reaping while
+ * suppressing their callbacks.
+ */
+
+struct sdxi_dma_chan {
+	struct virt_dma_chan vchan;
+	struct sdxi_cxt *cxt;
+	unsigned int vector;
+	unsigned int irq;
+	struct sdxi_akey_ent *akey;
+	/*
+	 * Transactions awaiting ring slots or completion blocks; protected by
+	 * vchan.lock.
+	 */
+	struct list_head pending;
+};
+
+struct sdxi_dma_txd {
+	struct virt_dma_desc vdesc;
+	struct sdxi_copy copy;	/* Copy operation parameters */
+	/* Completion ownership is acquired at issue and released after reaping. */
+	struct sdxi_completion completion;
+};
+
+static void sdxi_dma_reap_locked(struct sdxi_dma_chan *sdchan);
+
+struct sdxi_dma_dev {
+	struct dma_device dma_dev;
+	size_t nr_channels;
+	struct sdxi_dma_chan sdchan[] __counted_by(nr_channels);
+};
+
+static struct sdxi_dma_chan *to_sdxi_dma_chan(const struct dma_chan *dma_chan)
+{
+	const struct virt_dma_chan *vchan;
+
+	vchan = container_of_const(dma_chan, struct virt_dma_chan, chan);
+	return container_of(vchan, struct sdxi_dma_chan, vchan);
+}
+
+static struct sdxi_dma_txd *
+to_sdxi_dma_txd(const struct virt_dma_desc *vdesc)
+{
+	return container_of(vdesc, struct sdxi_dma_txd, vdesc);
+}
+
+static void sdxi_dma_txd_free(struct virt_dma_desc *vdesc)
+{
+	struct sdxi_dma_txd *txd = to_sdxi_dma_txd(vdesc);
+
+	kfree(txd);
+}
+
+static struct dma_async_tx_descriptor *
+sdxi_dma_prep_memcpy(struct dma_chan *dma_chan, dma_addr_t dst,
+		     dma_addr_t src, size_t len, unsigned long flags)
+{
+	struct sdxi_dma_chan *sdchan = to_sdxi_dma_chan(dma_chan);
+	struct sdxi_cxt *cxt = sdchan->cxt;
+	u16 akey_index = sdxi_akey_index(cxt, sdchan->akey);
+	struct sdxi_dma_txd *txd;
+	struct sdxi_copy copy = {
+		.src = src,
+		.dst = dst,
+		.src_akey = akey_index,
+		.dst_akey = akey_index,
+		.len = len,
+	};
+
+	/*
+	 * Validate now, with a throwaway encode onto the stack. The real
+	 * encode into the ring happens at issue.
+	 */
+	if (sdxi_encode_copy(&(struct sdxi_desc){}, &copy))
+		return NULL;
+
+	txd = kmalloc_obj(*txd, GFP_NOWAIT);
+	if (!txd)
+		return NULL;
+
+	*txd = (typeof(*txd)) {
+		.copy = copy,
+	};
+
+	return vchan_tx_prep(&sdchan->vchan, &txd->vdesc, flags);
+}
+
+static enum dma_status sdxi_tx_status(struct dma_chan *chan,
+				      dma_cookie_t cookie,
+				      struct dma_tx_state *state)
+{
+	struct sdxi_dma_chan *sdchan = to_sdxi_dma_chan(chan);
+	struct virt_dma_desc *vdesc;
+
+	/* Under the lock for a consistent cookie snapshot; the IRQ reaps. */
+	guard(spinlock_irqsave)(&sdchan->vchan.lock);
+
+	/* vchan_find_desc() only returns issued txds, which own a completion. */
+	vdesc = vchan_find_desc(&sdchan->vchan, cookie);
+	if (vdesc) {
+		struct sdxi_dma_txd *txd = to_sdxi_dma_txd(vdesc);
+
+		if (sdxi_cst_blk_check(txd->completion.cst_blk) == SDXI_COMPLETION_ERROR)
+			return DMA_ERROR;
+	}
+
+	return dma_cookie_status(chan, cookie, state);
+}
+
+/*
+ * Fill capacity with complete, fenced batches in cookie order. Return true
+ * if descriptors were published; the caller kicks once with the lock held.
+ */
+static bool sdxi_dma_drain_locked(struct sdxi_dma_chan *sdchan)
+{
+	struct sdxi_cxt *cxt = sdchan->cxt;
+	struct sdxi_ring_state *rs = cxt->ring_state;
+	bool pushed = false;
+
+	lockdep_assert_held(&sdchan->vchan.lock);
+
+	while (!list_empty(&sdchan->pending)) {
+		struct virt_dma_desc *vdesc;
+		struct sdxi_ring_resv resv;
+		struct sdxi_desc *desc;
+		size_t claimed = 0, copies, i = 0;
+
+		/* Claim a pending prefix, stopping at a flag boundary or capacity limit. */
+		list_for_each_entry(vdesc, &sdchan->pending, node) {
+			struct sdxi_dma_txd *txd = to_sdxi_dma_txd(vdesc);
+
+			if (!sdxi_cst_pool_get(cxt->cst_pool, &txd->completion))
+				break;
+			claimed++;
+			if (vdesc->tx.flags & (DMA_PREP_INTERRUPT | DMA_PREP_FENCE) ||
+			    claimed == rs->sq->count - 1)
+				break;
+		}
+		if (!claimed)
+			break;
+
+		/* Ring reservations cannot be undone; acquire completions first. */
+		copies = 0;
+		if (!sdxi_ring_try_reserve_range(rs, 2, claimed + 1, &resv))
+			copies = sdxi_ring_resv_count(&resv) - 1;
+
+		/* Return claims not covered by this (possibly failed) reservation. */
+		list_for_each_entry(vdesc, &sdchan->pending, node) {
+			struct sdxi_dma_txd *txd = to_sdxi_dma_txd(vdesc);
+
+			if (i == claimed)
+				break;
+			if (i++ < copies)
+				continue;
+			sdxi_cst_pool_put(cxt->cst_pool, txd->completion.cst_blk);
+			txd->completion = (struct sdxi_completion) {};
+		}
+		if (!copies)
+			break;
+		for (size_t i = 0; i < copies; i++) {
+			struct sdxi_dma_txd *txd;
+			struct sdxi_slot slot = sdxi_ring_resv_slot(&resv, i);
+
+			vdesc = list_first_entry(&sdchan->pending,
+						struct virt_dma_desc, node);
+			txd = to_sdxi_dma_txd(vdesc);
+			desc = sdxi_slot_desc(slot);
+			(void)sdxi_encode_copy(desc, &txd->copy); /* validated at prep */
+			sdxi_cxt_attach_cst(slot, &txd->completion);
+			sdxi_desc_make_valid(desc);
+			list_move_tail(&vdesc->node, &sdchan->vchan.desc_issued);
+		}
+
+		/*
+		 * FE waits for all copies before notifying and before parsing
+		 * subsequent copies (SDXI 1.0a 5.3, steps 7-9). One terminator
+		 * satisfies both flags and supplies internal completion IRQs.
+		 */
+		desc = sdxi_slot_desc(sdxi_ring_resv_slot(&resv, copies));
+		sdxi_encode_intr(desc, &(const struct sdxi_intr) {
+			.akey = sdxi_akey_index(cxt, sdchan->akey),
+		});
+		sdxi_desc_set_fence(desc);
+		sdxi_desc_make_valid(desc);
+		pushed = true;
+	}
+
+	return pushed;
+}
+
+static void sdxi_dma_issue_pending(struct dma_chan *dma_chan)
+{
+	struct sdxi_dma_chan *sdchan = to_sdxi_dma_chan(dma_chan);
+
+	guard(spinlock_irqsave)(&sdchan->vchan.lock);
+
+	list_splice_tail_init(&sdchan->vchan.desc_submitted, &sdchan->pending);
+	if (sdxi_dma_drain_locked(sdchan))
+		sdxi_cxt_kick(sdchan->cxt);
+}
+
+static int sdxi_dma_terminate_all(struct dma_chan *dma_chan)
+{
+	struct sdxi_dma_chan *sdchan = to_sdxi_dma_chan(dma_chan);
+	struct virt_dma_chan *vchan = &sdchan->vchan;
+	LIST_HEAD(head);
+
+	/*
+	 * Prepared descriptors remain caller-owned. Free submitted and pending
+	 * transactions, and move published transactions to desc_terminated in ring
+	 * order for callback-free reaping.
+	 */
+	scoped_guard(spinlock_irqsave, &vchan->lock) {
+		list_splice_tail_init(&sdchan->pending, &head);
+		list_splice_tail_init(&vchan->desc_submitted, &head);
+		list_splice_tail_init(&vchan->desc_issued, &vchan->desc_terminated);
+	}
+	vchan_dma_desc_free_list(vchan, &head);
+
+	return 0;
+}
+
+/* Bound the wait for issued and pending copies to complete. */
+#define SDXI_DMA_SYNC_TIMEOUT_US	(10 * USEC_PER_SEC)
+
+/*
+ * Reap without relying on the completion IRQ, and report whether the
+ * device is done with every txd that has been or may be placed in the
+ * ring. Pending txds count because dma_chan_put() synchronizes without
+ * terminating first.
+ */
+static bool sdxi_dma_idle(struct sdxi_dma_chan *sdchan)
+{
+	struct virt_dma_chan *vchan = &sdchan->vchan;
+
+	guard(spinlock_irqsave)(&vchan->lock);
+
+	sdxi_dma_reap_locked(sdchan);
+	if (sdxi_dma_drain_locked(sdchan))
+		sdxi_cxt_kick(sdchan->cxt);
+
+	return list_empty(&vchan->desc_terminated) &&
+	       list_empty(&vchan->desc_issued) &&
+	       list_empty(&sdchan->pending);
+}
+
+static void sdxi_dma_synchronize(struct dma_chan *dma_chan)
+{
+	struct sdxi_dma_chan *sdchan = to_sdxi_dma_chan(dma_chan);
+	struct virt_dma_chan *vchan = &sdchan->vchan;
+	LIST_HEAD(head);
+	bool idle;
+	int err;
+
+	/*
+	 * Callers cannot handle failure, so the wait is not interruptible.
+	 * Every completion was reaped under vchan.lock, so the tasklet that
+	 * runs its callback is already scheduled for vchan_synchronize().
+	 */
+	err = read_poll_timeout(sdxi_dma_idle, idle, idle, 100,
+				SDXI_DMA_SYNC_TIMEOUT_US, false, sdchan);
+	if (err) {
+		/*
+		 * Suppress callbacks for outstanding transactions. Completion blocks
+		 * not released by the reaper stay claimed until context teardown.
+		 */
+		scoped_guard(spinlock_irqsave, &vchan->lock) {
+			list_splice_tail_init(&vchan->desc_issued,
+					      &vchan->desc_terminated);
+			list_splice_tail_init(&sdchan->pending, &head);
+		}
+		vchan_dma_desc_free_list(vchan, &head);
+		dev_err(dma_chan->device->dev,
+			"timed out waiting for DMA channel %d to drain\n",
+			dma_chan->chan_id);
+	}
+
+	vchan_synchronize(vchan);
+}
+
+static void sdxi_dma_reap_locked(struct sdxi_dma_chan *sdchan)
+{
+	struct virt_dma_chan *vchan = &sdchan->vchan;
+	struct virt_dma_desc *vdesc, *tmp;
+
+	lockdep_assert_held(&vchan->lock);
+
+	/*
+	 * Terminated txds are still in the ring. Release them once the device
+	 * is done, without completing their cookies or invoking callbacks.
+	 */
+	list_for_each_entry_safe(vdesc, tmp, &vchan->desc_terminated, node) {
+		struct sdxi_dma_txd *txd = to_sdxi_dma_txd(vdesc);
+
+		if (sdxi_cst_blk_check(txd->completion.cst_blk) == SDXI_COMPLETION_PENDING)
+			break;
+
+		list_del(&vdesc->node);
+		sdxi_cst_pool_put(sdchan->cxt->cst_pool, txd->completion.cst_blk);
+		sdxi_dma_txd_free(vdesc);
+	}
+
+	/* Reap completed transactions in cookie order. */
+	while ((vdesc = vchan_next_desc(vchan))) {
+		struct sdxi_dma_txd *txd = to_sdxi_dma_txd(vdesc);
+		enum sdxi_completion_state state;
+
+		state = sdxi_cst_blk_check(txd->completion.cst_blk);
+		if (state == SDXI_COMPLETION_PENDING)
+			break;
+
+		if (state == SDXI_COMPLETION_ERROR)
+			txd->vdesc.tx_result.result = DMA_TRANS_ABORTED;
+
+		list_del(&vdesc->node);
+		sdxi_cst_pool_put(sdchan->cxt->cst_pool, txd->completion.cst_blk);
+		txd->completion = (struct sdxi_completion) {};
+		vchan_cookie_complete(&txd->vdesc);
+	}
+
+	sdxi_ring_wake_up(sdchan->cxt->ring_state);
+}
+
+static irqreturn_t sdxi_dma_cxt_irq(int irq, void *data)
+{
+	struct sdxi_dma_chan *sdchan = data;
+
+	guard(spinlock_irqsave)(&sdchan->vchan.lock);
+
+	sdxi_dma_reap_locked(sdchan);
+	/* Consumed completion blocks may let stalled pending txds proceed. */
+	if (sdxi_dma_drain_locked(sdchan))
+		sdxi_cxt_kick(sdchan->cxt);
+
+	return IRQ_HANDLED;
+}
+
+static int sdxi_dma_alloc_chan_resources(struct dma_chan *dma_chan)
+{
+	struct sdxi_dev *sdxi = dev_get_drvdata(dma_chan->device->dev);
+	struct sdxi_dma_chan *sdchan = to_sdxi_dma_chan(dma_chan);
+	int vector, irq, err;
+
+	sdchan->cxt = sdxi_cxt_new(sdxi);
+	if (!sdchan->cxt)
+		return -ENOMEM;
+	err = vector = sdxi_alloc_vector(sdxi);
+	if (vector < 0)
+		goto exit_cxt;
+
+	sdchan->vector = vector;
+
+	err = irq = sdxi_vector_to_irq(sdxi, vector);
+	if (irq < 0)
+		goto free_vector;
+
+	sdchan->irq = irq;
+
+	/*
+	 * Note this akey entry is used for both the completion
+	 * interrupt and source and destination access for copies.
+	 */
+	sdchan->akey = sdxi_alloc_akey(sdchan->cxt);
+	if (!sdchan->akey) {
+		err = -ENOMEM;
+		goto free_vector;
+	}
+
+	*sdchan->akey = (typeof(*sdchan->akey)) {
+		.intr_num = cpu_to_le16(FIELD_PREP(SDXI_AKEY_ENT_VL, 1) |
+					FIELD_PREP(SDXI_AKEY_ENT_IV, 1) |
+					FIELD_PREP(SDXI_AKEY_ENT_INTR_NUM,
+						   vector)),
+	};
+
+	err = request_irq(sdchan->irq, sdxi_dma_cxt_irq,
+			  IRQF_TRIGGER_NONE, "SDXI DMAengine", sdchan);
+	if (err)
+		goto free_akey;
+
+	err = sdxi_start_cxt(sdchan->cxt);
+	if (err)
+		goto free_irq;
+
+	return 0;
+free_irq:
+	free_irq(sdchan->irq, sdchan);
+free_akey:
+	sdxi_free_akey(sdchan->cxt, sdchan->akey);
+free_vector:
+	sdxi_free_vector(sdxi, vector);
+exit_cxt:
+	sdxi_cxt_exit(sdchan->cxt);
+	return err;
+}
+
+static void sdxi_dma_free_chan_resources(struct dma_chan *dma_chan)
+{
+	struct sdxi_dma_chan *sdchan = to_sdxi_dma_chan(dma_chan);
+	LIST_HEAD(head);
+
+	sdxi_stop_cxt(sdchan->cxt);
+	free_irq(sdchan->irq, sdchan);
+	sdxi_free_vector(sdchan->cxt->sdxi, sdchan->vector);
+	sdxi_free_akey(sdchan->cxt, sdchan->akey);
+	/* virt-dma does not know about the driver's pending list. */
+	scoped_guard(spinlock_irqsave, &sdchan->vchan.lock)
+		list_splice_tail_init(&sdchan->pending, &head);
+	vchan_dma_desc_free_list(&sdchan->vchan, &head);
+	vchan_free_chan_resources(to_virt_chan(dma_chan));
+	sdxi_cxt_exit(sdchan->cxt);
+}
+
+int sdxi_dma_register(struct sdxi_dev *sdxi)
+{
+	struct device *dev = sdxi->dev;
+	struct sdxi_dma_dev *sddev;
+	struct dma_device *dma_dev;
+	unsigned int nr_channels = dma_channels; /* writable module param; snapshot it */
+	int err;
+
+	if (!nr_channels)
+		return 0;
+	/*
+	 * TODO: Require IntrGrp support before registering the DMAengine provider
+	 * (SDXI 1.0a Table 6-1).
+	 */
+
+	sddev = devm_kzalloc(dev, struct_size(sddev, sdchan, nr_channels),
+			     GFP_KERNEL);
+	if (!sddev)
+		return -ENOMEM;
+
+	sddev->nr_channels = nr_channels;
+
+	dma_dev = &sddev->dma_dev;
+	*dma_dev = (typeof(*dma_dev)) {
+		.dev                 = dev,
+		.src_addr_widths     = DMA_SLAVE_BUSWIDTH_64_BYTES,
+		.dst_addr_widths     = DMA_SLAVE_BUSWIDTH_64_BYTES,
+		.directions          = BIT(DMA_MEM_TO_MEM),
+		.residue_granularity = DMA_RESIDUE_GRANULARITY_DESCRIPTOR,
+
+		.device_alloc_chan_resources = sdxi_dma_alloc_chan_resources,
+		.device_free_chan_resources  = sdxi_dma_free_chan_resources,
+
+		.device_prep_dma_memcpy = sdxi_dma_prep_memcpy,
+
+		.device_terminate_all = sdxi_dma_terminate_all,
+		.device_synchronize = sdxi_dma_synchronize,
+		.device_tx_status = sdxi_tx_status,
+		.device_issue_pending = sdxi_dma_issue_pending,
+	};
+
+	dma_cap_set(DMA_MEMCPY, dma_dev->cap_mask);
+	INIT_LIST_HEAD(&dma_dev->channels);
+
+	for (size_t i = 0; i < sddev->nr_channels; ++i) {
+		struct sdxi_dma_chan *sdchan = &sddev->sdchan[i];
+
+		sdchan->vchan.desc_free = sdxi_dma_txd_free;
+		vchan_init(&sdchan->vchan, &sddev->dma_dev);
+		INIT_LIST_HEAD(&sdchan->pending);
+	}
+
+	err = dmaenginem_async_device_register(dma_dev);
+	if (err)
+		return dev_warn_probe(dev, err, "failed to register dma device\n");
+
+	return 0;
+}
diff --git a/drivers/dma/sdxi/dma.h b/drivers/dma/sdxi/dma.h
new file mode 100644
index 000000000000..d38870ea7d91
--- /dev/null
+++ b/drivers/dma/sdxi/dma.h
@@ -0,0 +1,11 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright Advanced Micro Devices, Inc. */
+
+#ifndef DMA_SDXI_DMA_H
+#define DMA_SDXI_DMA_H
+
+struct sdxi_dev;
+
+int sdxi_dma_register(struct sdxi_dev *sdxi);
+
+#endif /* DMA_SDXI_DMA_H */

-- 
2.56.0

Re: [PATCH v4 07/25] dmaengine: sdxi: Allocate context control DMA pool

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> Each SDXI context has a control block (CXT_CTL) that identifies its
> descriptor ring, context status block, and write index.
> 
> Define the CXT_CTL layout and create a DMA pool for them during
> function initialization.
> 
> Co-developed-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=7

Re: [PATCH v4 05/25] dmaengine: sdxi: Feature discovery and initial configuration

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> Discover the stopped function's version, doorbell stride, supported
> operation groups, and context and buffer limits. Apply the driver
> configuration to MMIO_CTL2 after function preparation has removed
> inherited state.
> 
> Add capability fields and device bookkeeping. Report the SDXI version implemented by the function.
> 
> Co-developed-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=5

Re: [PATCH v4 06/25] dmaengine: sdxi: Configure context tables

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> SDXI uses a two-level table hierarchy to track contexts. There is a
> single level 2 table per function which enumerates up to 512 level 1
> tables. Each level 1 table enumerates up to 128 contexts.
> 
> Allocate and install the L2 table and a single L1 table, enough for
> context IDs 0-127 (i.e. the admin context with reserved id 0, plus 127
> client contexts). For now, to avoid dynamic management of additional
> L1 tables, cap ctl2.max_cxt to 127.
> 
> Since the table allocations are devres-managed, there is no
> corresponding cleanup code required.
> 
> Co-developed-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=6

Re: [PATCH v4 01/25] PCI: Add SNIA SDXI accelerator sub-class

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> Make the class code for SNIA Smart Data Accelerator Interface (SDXI)
> functions available to both C and Rust code.
> 
> See PCI Code and ID Assignment spec r1.14, sec 1.19.
> 
> Co-developed-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=1

Re: [PATCH v4 09/25] dmaengine: sdxi: Allocate administrative context

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> Allocate the administrative context’s control structures and register
> device-managed cleanup.
> 
> Co-developed-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=9

Re: [PATCH v4 08/25] dmaengine: sdxi: Introduce submission queue

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> Add the sdxi_sq (submission queue) type and APIs that act on it. The
> sq contains three control structures shared with hardware: descriptor
> ring, write index, and context status block (which contains the read
> index). Define the hardware context-status and generic descriptor
> layouts, including assertions for their sizes.
> 
> The descriptor ring requires only 64-byte alignment, so it's practical
> to implement it as a flexible array member. The write index and context
> status are small enough to place in the struct as well, so we have a
> single DMA coherent allocation to manage.
> 
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=8

Re: [PATCH v4 03/25] dmaengine: sdxi: Add PCI initialization

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> Add enough code to bind a SDXI device via the class code and map its
> control registers and doorbell region.
> 
> Keep PCI resource setup in pci.c and register devices with the SDXI core
> through bus operations.
> 
> Co-developed-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=3

Re: [PATCH v4 10/25] dmaengine: sdxi: Install administrative context

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> Serialize the context control block, akey table, and L1 entry for the
> admin context, making its descriptor ring, write index, and context
> status block visible to the SDXI implementation once it is activated.
> 
> Co-developed-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=10

Re: [PATCH v4 15/25] dmaengine: sdxi: Per-context access key (AKey) table entry allocator

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> Each SDXI context has a table of access keys (AKeys). SDXI descriptors
> submitted to a context may refer to an AKey associated with that
> context by its index in the table. AKeys describe properties of the
> access that the descriptor is to perform, such as PASID or a target
> SDXI function, or an interrupt to trigger.
> 
> Use a per-context IDA to keep track of used entries in the table.
> Provide sdxi_alloc_akey(), which claims an AKey table entry for the
> caller to program directly; sdxi_akey_index(), which returns the
> entry's index for programming into descriptors the caller intends to
> submit; and sdxi_free_akey(), which clears the entry and makes it
> available again.
> 
> The DMAengine provider uses one AKey for copy-buffer access and completion
> interrupts.
> 
> Co-developed-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=15

Re: [PATCH v4 16/25] dmaengine: sdxi: Add completion status block API

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> Introduce completion status handling and a fixed-capacity pool owned by
> each context. Only the 32-byte-aligned status blocks are DMA-visible;
> a private bitmap and spinlock track claims independently of descriptor
> ring indices.
> 
> Preallocate storage during context allocation. Claiming and initializing
> a block cannot allocate memory and fails only when capacity is exhausted.
> Provide a bounded, killable wait for administrative callers, and wake
> waiters when blocks are returned. Release blocks only after consumption,
> unpublished rollback, or hardware quiescence; destroy the pool only when
> hardware and software no longer access it.
> 
> Represent a completion-status block by its CPU pointer and DMA address.
> Check the little-endian signal with READ_ONCE(), then use dma_rmb() before
> consuming error status or operation results. Polling returns success,
> -EIO for a terminal error, or -ETIMEDOUT without relinquishing ownership.
> 
> Co-developed-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=16

Re: [PATCH v4 17/25] dmaengine: sdxi: Add unit tests for completion pool ownership

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> Add hardware-independent KUnit coverage for completion-pool exhaustion
> across a bitmap word boundary, arbitrary release and reuse, reinitialization
> of reused status blocks. Verify that a claimed block remains unavailable
> until explicitly released, including after a simulated late device completion.
> Also cover release of an unpublished claim.
> 
> Provide test allocation helpers and KUnit-only exports so the tests exercise
> the production claim and release paths without a coherent-DMA device.
> 
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=17

Re: [PATCH v4 18/25] dmaengine: sdxi: Add context submission helper

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> Add sdxi_cxt_submit() to consolidate optional fence and completion-status
> block setup, valid-bit publication, and doorbell ringing. An optional
> struct sdxi_completion input identifies storage already claimed and
> initialized by the caller.
> 
> Introduce generic descriptor helpers for attaching completion storage,
> setting completion-status mode, fencing, and publishing the valid bit.
> Publishing orders prior descriptor writes before transferring ownership
> to hardware; the helpers warn if a caller attempts to modify an already
> valid descriptor.
> 
> Provide sdxi_cxt_attach_cst() for attaching completion storage without
> publishing the descriptor, and sdxi_cxt_kick() for ringing a context
> doorbell.
> 
> Co-developed-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=18

Re: [PATCH v4 19/25] dmaengine: sdxi: Encode context start, stop, update, and sync descriptors

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> Add layouts and encoders for administrative context start, stop, update,
> and synchronization descriptors. Parameters select the context range,
> update level, and synchronization filter. Encoders set mandatory fields
> and produce little-endian descriptors with VL clear; callers attach
> completion storage and publish them.
> 
> Co-developed-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=19

Re: [PATCH v4 12/25] dmaengine: sdxi: Add client context allocation API

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> Allocate and publish client contexts, track their IDs in a per-device
> xarray, and release them during device teardown.
> 
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=12

Re: [PATCH v4 14/25] dmaengine: sdxi: Add unit tests for descriptor ring reservations

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] dmaengine: sdxi: unused mock KUnit device in setup_device()
--

commit 73b5f711479c1dbab69a55b0e9eb01abd747309e
Author: Nathan Lynch <nathan.lynch@amd.com>

dmaengine: sdxi: Add unit tests for descriptor ring reservations

This commit adds hardware-independent KUnit tests for SDXI descriptor ring
reservations, covering full-ring and single-slot reservations, invalid counts,
inconsistent indices, insufficient space, and bounded reservations across ring
wrap.

> diff --git a/drivers/dma/sdxi/ring_kunit.c b/drivers/dma/sdxi/ring_kunit.c
> new file mode 100644
> index 0000000000000..7df85fc5549c6
> --- /dev/null
> +++ b/drivers/dma/sdxi/ring_kunit.c
> @@ -0,0 +1,147 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * SDXI descriptor ring management tests.
> + *
> + * Copyright Advanced Micro Devices, Inc.
> + */
> +#include <kunit/device.h>
> +#include <kunit/test-bug.h>
> +#include <kunit/test.h>
> +#include <linux/container_of.h>
> +#include <linux/dma-mapping.h>
> +#include <linux/module.h>
> +#include <linux/overflow.h>
> +#include <linux/packing.h>
> +#include <linux/string.h>
> +
> +#include "ring.h"
> +#include "submission.h"
> +
> +MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
> +
> +/* A submission queue backed by plain memory; no device or DMA required. */
> +static struct sdxi_sq *alloc_sq(struct kunit *t, u32 count)
> +{
> +	struct sdxi_sq *sq;
> +
> +	sq = kunit_kmalloc(t, struct_size(sq, ring, count),
> +			   GFP_KERNEL | __GFP_ZERO);
> +	if (sq)
> +		sq->count = count;
> +	return sq;
> +}

[ ... ]

> +static int setup_device(struct kunit *t)
> +{
> +	struct device *dev = kunit_device_register(t, "sdxi-mock-device");
> +
> +	KUNIT_ASSERT_NOT_ERR_OR_NULL(t, dev);
> +	t->priv = dev;
> +	return 0;
> +}
> +
> +static struct kunit_suite generic_desc_ts = {
> +	.name = "SDXI descriptor ring management",
> +	.test_cases = testcases,
> +	.init = setup_device,
> +};

[Severity: Low]
As noted in the comment above alloc_sq(), the submission queue is backed by
plain memory and does not require a struct device or DMA.

However, generic_desc_ts wires up setup_device() as its .init callback, which
registers "sdxi-mock-device" via kunit_device_register() and stores it in
t->priv before each test case (valid(), invalid(), and
bounded_reservations()), even though nothing in ring_kunit.c ever reads
t->priv.

Is setup_device() needed here, or can it (along with the unused
kunit/device.h, linux/container_of.h, linux/dma-mapping.h, linux/packing.h,
and linux/string.h includes) be dropped?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=14

Re: [PATCH v4 11/25] dmaengine: sdxi: Start functions on probe, stop on remove

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] dmaengine: sdxi: PF stopped before pci_disable_sriov() in sdxi_pci_remove()
--

commit 44d5d7973859c1db8741ba90d53413150ded9a60
Author: Nathan Lynch <nathan.lynch@amd.com>

dmaengine: sdxi: Start functions on probe, stop on remove

This commit activates the SDXI function during device registration by
setting MMIO_CTL0.fn_gsr to GSRV_ACTIVE and polling for GSV_ACTIVE or
GSV_ERROR, and stops the function via sdxi_unregister() on PCI removal.

> diff --git a/drivers/dma/sdxi/pci.c b/drivers/dma/sdxi/pci.c
> index 6a293db69199f..d196ab5cf6f1c 100644
> --- a/drivers/dma/sdxi/pci.c
> +++ b/drivers/dma/sdxi/pci.c
> @@ -99,6 +99,7 @@ static int sdxi_pci_probe(struct pci_dev *pdev,
>  
>  static void sdxi_pci_remove(struct pci_dev *pdev)
>  {
> +	sdxi_unregister(&pdev->dev);
>  	pci_disable_sriov(pdev);
>  }

[Severity: High]
Should pci_disable_sriov(pdev) be called before sdxi_unregister(&pdev->dev)
in sdxi_pci_remove()?

When SR-IOV virtual functions have been enabled via sriov_configure after PF
registration, calling sdxi_unregister() first invokes sdxi_dev_stop() on the
PF while the child VFs are still active and bound to their driver instances:

sdxi_pci_remove()
  sdxi_unregister()
    sdxi_dev_stop()
      sdxi->bus_ops->reset() -> sdxi_pci_reset()
        pci_reset_function_locked()

If the native MMIO stop sequence fails or times out, sdxi_dev_stop() falls
back to sdxi_pci_reset() and issues a PF function-level reset (or clears bus
mastering). A PF FLR clears PCI_SRIOV_CTRL_VFE in hardware and resets all VFs
underneath their bound drivers, and pci_dev_restore() -> sriov_restore_state()
then re-enables PCI_SRIOV_CTRL_VFE with uninitialized post-reset VFs before
pci_disable_sriov(pdev) runs to unbind and remove them.

Calling pci_disable_sriov(pdev) before sdxi_unregister(&pdev->dev) would
ensure all child VFs are unbound, stopped, and disabled before the PF itself
is stopped or reset.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=11

Re: [PATCH v4 21/25] dmaengine: sdxi: Provide context start and stop APIs

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> Implement client context start and stop by submitting administrative
> descriptors. Starting waits for the start descriptor to finish; stopping
> submits stop followed by synchronization and waits for the latter.
> 
> Use the administrative reservation and polling helpers to claim status
> blocks before reserving descriptors, release consumed results, and retain
> timed-out blocks until context teardown. Expose these operations through
> struct sdxi_cxt for clients such as the DMAengine provider.
> 
> Co-developed-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=21

Re: [PATCH v4 23/25] dmaengine: sdxi: Add unit tests for descriptor encoding

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> Test the encoder function for each descriptor type the driver encodes.
> 
> Decode descriptors with the packing API using independently specified
> bit offsets, so errors in the production bitfield masks can be detected.
> 
> Co-developed-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=23

Re: [PATCH v4 04/25] dmaengine: sdxi: Prepare function and sanitize MMIO state

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> Recover an inherited function state using bounded native stop/reset
> transitions and a PCI function reset that saves and restores PCI
> configuration. Use the locked reset API because probe and removal hold
> the device lock. If reset fails, disable bus mastering and wait
> for pending transactions as a best-effort fallback.
> 
> Disable inherited error logging before stop commands can provoke errors.
> After stopping, sanitize writable configuration in Chapter 9 order:
> preserve the function group ID, clear PASID and interrupt enables,
> restore CTL2 reset limits with operation groups disabled, discard stale
> context and supported RKey tables, and clear error-log state. Disable
> mailbox interrupts only on functions with SR-IOV mailbox support.
> 
> Keep mailbox commands and peer-owned payloads untouched. Status cleanup
> uses the specified write-one-to-clear semantics. Invoke preparation
> before feature configuration or publishing new DMA control structures.
> 
> Co-developed-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=4

Re: [PATCH v4 22/25] dmaengine: sdxi: Encode copy and interrupt descriptors

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> Add copy and interrupt descriptor layouts and encoders. Both encoders
> accept operation-specific parameter structs.
> 
> Copy descriptors are used to implement memcpy offload for the DMA
> engine provider, and interrupt descriptors are used to signal the
> completion of preceding descriptors in the ring.
> 
> Conditionally expose sdxi_encode_size32() for unit testing.
> 
> Co-developed-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=22

Re: [PATCH v4 13/25] dmaengine: sdxi: Add descriptor ring management

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> Introduce software state and reservation helpers for driving an SDXI
> submission queue. A spinlock serializes space checks and write-index
> updates; blocking reservations use a bounded, killable wait.
> 
> Provide exact-size and bounded-size reservations, with descriptor access
> through a ring slot carrying its index and ring state. Reservations
> publish the write index and cannot be rolled back, so callers must
> populate and publish every reserved slot.
> 
> Allocate and initialize ring state with each administrative or client
> context, and free it during context release.
> 
> The hardware read index bounds descriptor reuse. Completion handlers
> wake blocked reservations as hardware consumes slots.
> 
> Co-developed-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Wei Huang <wei.huang2@amd.com>
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=13

Re: [PATCH v4 20/25] dmaengine: sdxi: Add client context release with update and sync

[sashiko-bot]

> From: Nathan Lynch <nathan.lynch@amd.com>
> 
> Invalidate an unused, stopped client context and synchronize hardware
> caches before freeing its resources. Add administrative update, reservation, and
> completion-polling helpers.
> 
> Claim completion-status storage before making an irrevocable ring
> reservation. Return the claim if reservation fails, and release it after
> consuming a terminal result. A timeout leaves hardware ownership
> unresolved, so retain that block in the administrative pool until context
> teardown, even if the device writes its terminal signal later. Repeated
> timeouts can exhaust the pool, causing further claims to time out rather
> than reuse outstanding status blocks.
> 
> If update or synchronization fails, retain the context and its ID for
> cleanup after function shutdown. Device-wide teardown frees clients after
> the best-effort function stop/reset path without submitting additional
> administrative operations.
> 
> Signed-off-by: Nathan Lynch <nathan.lynch@amd.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=20

Re: [PATCH v4 24/25] dmaengine: sdxi: MSI/MSI-X vector allocation and mapping

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] dmaengine: sdxi: incomplete kernel-doc comments in sdxi.h
--

commit fccd22f95bf4bdc550267ac2709b80e00e1e9267
Author: Nathan Lynch <nathan.lynch@amd.com>

dmaengine: sdxi: MSI/MSI-X vector allocation and mapping

This commit allocates MSI/MSI-X vectors for the SDXI error log interrupt
and client contexts, tracking non-reserved vectors with an IDA. It also
adds helpers to allocate, free, and map device-relative vector indices to
Linux IRQ numbers via a new get_irq bus operation.

> diff --git a/drivers/dma/sdxi/sdxi.h b/drivers/dma/sdxi/sdxi.h
> index aaa43eab00b33..07044d837d9af 100644
> --- a/drivers/dma/sdxi/sdxi.h
> +++ b/drivers/dma/sdxi/sdxi.h

[ ... ]

> @@ -69,6 +90,38 @@ struct sdxi_dev {
>  	const struct sdxi_bus_ops *bus_ops;
>  };
>  
> +/**
> + * sdxi_alloc_vector() - Allocate an interrupt vector.
> + *
> + * A vector that will have the same lifetime as the device does not
> + * need to be released explicitly. Otherwise the vector must be
> + * released with sdxi_free_vector().
> + */
> +static inline int sdxi_alloc_vector(struct sdxi_dev *sdxi)
> +{
> +	return ida_alloc_range(&sdxi->vectors, SDXI_MIN_VECTORS,
> +			       sdxi->nr_vectors - 1, GFP_KERNEL);
> +}
> +
> +/**
> + * sdxi_free_vector() - Release a previously allocated index.
> + */
> +static inline void sdxi_free_vector(struct sdxi_dev *sdxi, unsigned int nr)
> +{
> +	ida_free(&sdxi->vectors, nr);
> +}
> +
> +/**
> + * sdxi_vector_to_irq() - Translate an allocated interrupt vector to
> + *                        Linux IRQ number suitable for passing to
> + *                        request_irq() et al.
> + */
> +static inline int sdxi_vector_to_irq(struct sdxi_dev *sdxi, unsigned int nr)
> +{
> +	WARN_ON_ONCE(!ida_exists(&sdxi->vectors, nr));
> +	return sdxi->bus_ops->get_irq(sdxi, nr);
> +}

[Severity: Low]
sdxi_alloc_vector(), sdxi_free_vector(), and sdxi_vector_to_irq() in
drivers/dma/sdxi/sdxi.h use the /** kernel-doc comment marker without
describing the @sdxi and @nr parameters or including Return: sections for
sdxi_alloc_vector() and sdxi_vector_to_irq().

Running scripts/kernel-doc on drivers/dma/sdxi/sdxi.h (for example via
make HEADER_CHECK=drivers/dma/sdxi headercheck) emits kernel-doc warnings
for the undescribed parameters and missing return value descriptions.

Should these comments document @sdxi, @nr, and Return:, or use regular /*
comments instead?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=24

Re: [PATCH v4 25/25] dmaengine: sdxi: Add DMA engine provider

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 7 potential issue(s) to consider:
- [High] dmaengine: sdxi: devm DMA unregister runs after sdxi_unregister() frees cxts
- [High] dmaengine: sdxi: sdxi_tx_status() loses error state and skips dma_tx_state
- [Medium] dmaengine: sdxi: sdxi_device_init() ignores sdxi_dma_register() error
- [Medium] dmaengine: sdxi: context stop-on-error prevents IRQ reaping and stalls sync
- [Medium] dmaengine: sdxi: premature AKey and vector release on context teardown
- [Medium] dmaengine: sdxi: missing max_buffer check in sdxi_dma_prep_memcpy()
- [Low] dmaengine: sdxi: undocumented dma_channels parameter in kernel-parameters.txt
--

commit cdf3a67512f9931ad28471e39e5814ce577c3d81
Author: Nathan Lynch <nathan.lynch@amd.com>

dmaengine: sdxi: Add DMA engine provider

This commit registers a virt-dma memcpy provider for SDXI functions with a
configurable number of channels per function. It batches copy descriptors
behind fenced completion interrupts and implements channel termination and
synchronization callbacks.

> diff --git a/drivers/dma/sdxi/device.c b/drivers/dma/sdxi/device.c
> index 30c4e687cedf1..265ebea1e350e 100644
> --- a/drivers/dma/sdxi/device.c
> +++ b/drivers/dma/sdxi/device.c
> @@ -23,6 +23,7 @@
>  #include <linux/xarray.h>
>  
>  #include "context.h"
> +#include "dma.h"
>  #include "hw.h"
>  #include "mmio.h"
>  #include "sdxi.h"
> @@ -424,6 +425,7 @@ static int sdxi_device_init(struct sdxi_dev *sdxi)
>  	if (err)
>  		return err;
>  
> +	sdxi_dma_register(sdxi);
>  	return 0;
>  }

[Severity: Medium]
As reported by Sashiko on v3 [1], sdxi_dma_register() returns -ENOMEM when
devm_kzalloc() fails and returns the negative error code from
dmaenginem_async_device_register() when registration fails, but
sdxi_device_init() ignores the return value and unconditionally returns 0.

Should sdxi_device_init() check the return value of sdxi_dma_register() and
clean up via sdxi_dev_stop() on failure, or should sdxi_dma_register() return
void if DMA engine registration is intentionally best-effort?

> diff --git a/drivers/dma/sdxi/dma.c b/drivers/dma/sdxi/dma.c
> new file mode 100644
> index 0000000000000..512855c83da27
> --- /dev/null
> +++ b/drivers/dma/sdxi/dma.c
> @@ -0,0 +1,509 @@

[ ... ]

> +static unsigned short dma_channels = 1;
> +module_param(dma_channels, ushort, 0644);
> +MODULE_PARM_DESC(dma_channels, "DMA channels per function (default: 1)");

[Severity: Low]
This adds the sdxi_core.dma_channels module parameter, which can also be set
on the kernel command line at boot. Should this parameter be documented in
Documentation/admin-guide/kernel-parameters.txt?

[ ... ]

> +static struct dma_async_tx_descriptor *
> +sdxi_dma_prep_memcpy(struct dma_chan *dma_chan, dma_addr_t dst,
> +		     dma_addr_t src, size_t len, unsigned long flags)
> +{
> +	struct sdxi_dma_chan *sdchan = to_sdxi_dma_chan(dma_chan);
> +	struct sdxi_cxt *cxt = sdchan->cxt;
> +	u16 akey_index = sdxi_akey_index(cxt, sdchan->akey);
> +	struct sdxi_dma_txd *txd;
> +	struct sdxi_copy copy = {
> +		.src = src,
> +		.dst = dst,
> +		.src_akey = akey_index,
> +		.dst_akey = akey_index,
> +		.len = len,
> +	};
> +
> +	/*
> +	 * Validate now, with a throwaway encode onto the stack. The real
> +	 * encode into the ring happens at issue.
> +	 */
> +	if (sdxi_encode_copy(&(struct sdxi_desc){}, &copy))
> +		return NULL;

[Severity: Medium]
In sdxi_fn_activate() and sdxi_publish_cxt(), the function and client context
are configured with sdxi->max_buffer, which restricts each copy descriptor to
at most (u64)SZ_2M << sdxi->max_buffer bytes:

drivers/dma/sdxi/device.c:sdxi_fn_activate()
	sdxi->max_buffer = FIELD_GET(SDXI_MMIO_CAP1_MAX_BUFFER, cap1);
	dma_set_max_seg_size(sdxi->dev,
			     min((u64)SZ_2M << sdxi->max_buffer, U32_MAX));

drivers/dma/sdxi/context.c:sdxi_publish_cxt()
	*ent = (typeof(*ent)) {
		...
		.max_buffer  = cxt->sdxi->max_buffer,
		...
	};

However, sdxi_dma_prep_memcpy() only validates copy.len through
sdxi_encode_copy() -> sdxi_encode_size32(), which accepts any length up to
SZ_4G. Because DMA engine memcpy callers such as dmatest and async_memcpy do
not check dma_get_max_seg_size() before calling device_prep_dma_memcpy(), can
a transfer with len > ((u64)SZ_2M << cxt->sdxi->max_buffer) succeed in
sdxi_dma_prep_memcpy() and then fault the hardware context when issued?

> +
> +	txd = kmalloc_obj(*txd, GFP_NOWAIT);
> +	if (!txd)
> +		return NULL;
> +
> +	*txd = (typeof(*txd)) {
> +		.copy = copy,
> +	};
> +
> +	return vchan_tx_prep(&sdchan->vchan, &txd->vdesc, flags);
> +}
> +
> +static enum dma_status sdxi_tx_status(struct dma_chan *chan,
> +				      dma_cookie_t cookie,
> +				      struct dma_tx_state *state)
> +{
> +	struct sdxi_dma_chan *sdchan = to_sdxi_dma_chan(dma_chan);
> +	struct virt_dma_desc *vdesc;
> +
> +	/* Under the lock for a consistent cookie snapshot; the IRQ reaps. */
> +	guard(spinlock_irqsave)(&sdchan->vchan.lock);
> +
> +	/* vchan_find_desc() only returns issued txds, which own a completion. */
> +	vdesc = vchan_find_desc(&sdchan->vchan, cookie);
> +	if (vdesc) {
> +		struct sdxi_dma_txd *txd = to_sdxi_dma_txd(vdesc);
> +
> +		if (sdxi_cst_blk_check(txd->completion.cst_blk) == SDXI_COMPLETION_ERROR)
> +			return DMA_ERROR;
> +	}
> +
> +	return dma_cookie_status(chan, cookie, state);
> +}

[Severity: High]
As reported by Sashiko on v3 [2], when a descriptor finishes with
SDXI_COMPLETION_ERROR, sdxi_dma_reap_locked() removes it from
vchan->desc_issued and calls vchan_cookie_complete():

	if (state == SDXI_COMPLETION_ERROR)
		txd->vdesc.tx_result.result = DMA_TRANS_ABORTED;

	list_del(&vdesc->node);
	sdxi_cst_pool_put(sdchan->cxt->cst_pool, txd->completion.cst_blk);
	txd->completion = (struct sdxi_completion) {};
	vchan_cookie_complete(&txd->vdesc);

Because vchan_cookie_complete() advances chan->completed_cookie to that
descriptor's cookie and vchan_find_desc() only searches vchan->desc_issued,
won't any call to sdxi_tx_status() after the descriptor is reaped fall through
to dma_cookie_status() and report DMA_COMPLETE instead of DMA_ERROR?

Also, when vchan_find_desc() does find an errored descriptor still on
vchan->desc_issued, returning DMA_ERROR directly skips dma_cookie_status()
and leaves *state uninitialized when the caller passes a non-NULL state
pointer (such as in dma_async_is_tx_complete()).

[ ... ]

> +static bool sdxi_dma_idle(struct sdxi_dma_chan *sdchan)
> +{
> +	struct virt_dma_chan *vchan = &sdchan->vchan;
> +
> +	guard(spinlock_irqsave)(&vchan->lock);
> +
> +	sdxi_dma_reap_locked(sdchan);
> +	if (sdxi_dma_drain_locked(sdchan))
> +		sdxi_cxt_kick(sdchan->cxt);
> +
> +	return list_empty(&vchan->desc_terminated) &&
> +	       list_empty(&vchan->desc_issued) &&
> +	       list_empty(&sdchan->pending);
> +}

[Severity: Medium]
Client contexts are configured with stop-on-error enabled in
sdxi_publish_cxt():

drivers/dma/sdxi/context.c:sdxi_publish_cxt()
	ctl_cfg = (typeof(ctl_cfg)) {
		.se              = 1,
		.csa             = 1,
		...
	};

In sdxi_dma_drain_locked(), completion interrupts are generated by a fenced
DSC_INTR descriptor placed at the end of each copy batch. If a copy descriptor
encounters an error, the context stops on that descriptor before executing the
trailing fenced DSC_INTR descriptor or any subsequent copies in the ring.

When that happens, won't the completion interrupt never fire, and won't
sdxi_dma_idle() in sdxi_dma_synchronize() continue waiting on the remaining
issued and pending descriptors until the 10-second SDXI_DMA_SYNC_TIMEOUT_US
expires rather than detecting that cxt->sq->cxt_sts.state has entered
CXTV_STOP_FN or CXTV_ERR_FN?

[ ... ]

> +static int sdxi_dma_alloc_chan_resources(struct dma_chan *dma_chan)
> +{

[ ... ]

> +	err = sdxi_start_cxt(sdchan->cxt);
> +	if (err)
> +		goto free_irq;
> +
> +	return 0;
> +free_irq:
> +	free_irq(sdchan->irq, sdchan);
> +free_akey:
> +	sdxi_free_akey(sdchan->cxt, sdchan->akey);
> +free_vector:
> +	sdxi_free_vector(sdxi, vector);
> +exit_cxt:
> +	sdxi_cxt_exit(sdchan->cxt);
> +	return err;
> +}
> +
> +static void sdxi_dma_free_chan_resources(struct dma_chan *dma_chan)
> +{
> +	struct sdxi_dma_chan *sdchan = to_sdxi_dma_chan(dma_chan);
> +	LIST_HEAD(head);
> +
> +	sdxi_stop_cxt(sdchan->cxt);
> +	free_irq(sdchan->irq, sdchan);
> +	sdxi_free_vector(sdchan->cxt->sdxi, sdchan->vector);
> +	sdxi_free_akey(sdchan->cxt, sdchan->akey);
> +	/* virt-dma does not know about the driver's pending list. */
> +	scoped_guard(spinlock_irqsave, &sdchan->vchan.lock)
> +		list_splice_tail_init(&sdchan->pending, &head);
> +	vchan_dma_desc_free_list(&sdchan->vchan, &head);
> +	vchan_free_chan_resources(to_virt_chan(dma_chan));
> +	sdxi_cxt_exit(sdchan->cxt);
> +}

[Severity: Medium]
In sdxi_dma_free_chan_resources(), if sdxi_stop_cxt() fails or times out (or
is interrupted in sdxi_cxt_reserve()), or if sdxi_cxt_update() fails inside
sdxi_cxt_exit(), sdxi_cxt_exit() retains the context until function shutdown
because the hardware may still reference it:

drivers/dma/sdxi/context.c:sdxi_cxt_exit()
	state = FIELD_GET(SDXI_CXT_STS_STATE, READ_ONCE(cxt->sq->cxt_sts.state));
	if (state != CXTV_STOP_SW && state != CXTV_STOP_FN && state != CXTV_ERR_FN)
		return -EBUSY;
	...
	err = sdxi_cxt_update(cxt);
	if (err) {
		dev_err(sdxi->dev,
			"context %u release failed (%d), retaining resources until function shutdown\n",
			cxt->id, err);
		return err;
	}

	sdxi_free_cxt(cxt);

Because sdxi_dma_free_chan_resources() calls sdxi_free_vector() and
sdxi_free_akey() unconditionally before sdxi_cxt_exit(), can this zero the
DMA-visible AKey entry and release sdchan->vector for reuse by another
channel while the unstopped context is still active in hardware?

Similarly, in sdxi_dma_alloc_chan_resources(), if sdxi_start_cxt() times out
after publishing DSC_CXT_START to the admin ring, jumping to free_irq frees
the AKey and vector and calls sdxi_cxt_exit() without stopping the context,
while DSC_CXT_START may still be pending or may have just transitioned the
context to CXTV_RUN.

> +
> +int sdxi_dma_register(struct sdxi_dev *sdxi)
> +{

[ ... ]

> +	err = dmaenginem_async_device_register(dma_dev);
> +	if (err)
> +		return dev_warn_probe(dev, err, "failed to register dma device\n");
> +
> +	return 0;
> +}

[Severity: High]
Does using dmaenginem_async_device_register() here cause a use-after-free and
double-free during driver unbind or PCI removal?

During probe, sdxi_pci_probe() -> sdxi_register() -> sdxi_device_init() ->
sdxi_dma_register() registers dma_dev with dmaenginem_async_device_register(),
which defers dma_async_device_unregister() to devres_release_all() after the
bus remove callback returns.

When the device is removed or unbound, sdxi_pci_remove() calls
sdxi_unregister() first:

drivers/dma/sdxi/device.c:sdxi_unregister()
	sdxi_dev_stop(sdxi);
	sdxi_client_cxts_exit(sdxi);
	ida_destroy(&sdxi->vectors);

sdxi_client_cxts_exit() frees every client context in sdxi->client_cxts via
sdxi_free_cxt() and destroys the xarray, while dma_dev and any allocated
channels are still registered. When devres_release_all() subsequently runs
dmaenginem_async_device_unregister() -> dma_async_device_unregister() ->
sdxi_dma_free_chan_resources() (or if an interrupt or channel callback runs
before devres teardown), sdxi_dma_free_chan_resources() dereferences the
already-freed sdchan->cxt, accesses the destroyed sdxi->vectors IDA and
sdxi->client_cxts xarray, and frees sdchan->cxt a second time in
sdxi_cxt_exit().

[1] https://lore.kernel.org/r/20260605-sdxi-base-v3-23-4d38ca2bdffe@amd.com
[2] https://lore.kernel.org/r/20260605-sdxi-base-v3-23-4d38ca2bdffe@amd.com

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261103-sdxi-base-v4-0-3668ccc7aab8@amd.com?part=25